text:
\section{Introduction}\label{sec:intro}

\begin{figure}[t]
\centering
\includegraphics[trim={2.6cm 8cm 2.7cm 7.7cm},clip, width=\columnwidth]{figures/numeracy_plot.pdf}
  \caption{We train a probing model to decode a number from its word embedding over a random 80\% of the integers from [-500, 500], e.g., ``71'' $\to$ 71.0. We plot the model's predictions for all numbers from [-2000, 2000]. The model accurately decodes numbers within the training range (in \textcolor{blue}{blue}), i.e., pre-trained embeddings like GloVe and BERT capture numeracy. However, the probe fails to extrapolate to larger numbers (in \textcolor{red}{red}). The Char-CNN (e) and Char-LSTM (f)  are trained jointly with the probing model.}
  \label{fig:plot}
\end{figure}

Neural NLP models have become the de-facto standard tool across language understanding tasks, even solving basic reading comprehension and textual entailment datasets~\cite{yu2018qanet,devlin2018BERT}. Despite this, existing models are incapable of complex forms of reasoning, in particular, we focus on the ability to reason \emph{numerically}. Recent datasets such as DROP{}~\cite{drop}, EQUATE{}~\cite{ravichander2019equate}, or Mathematics Questions~\cite{saxton2019analysing} test numerical reasoning; they contain examples which require comparing, sorting, and adding numbers in natural language (e.g., Figure~\ref{fig:example_1}).

The first step in performing numerical reasoning over natural language is \emph{numeracy}: the ability to understand and work with numbers in either digit or word form~\cite{Spithourakis2018numeracy}. For example, one must understand that the string ``23'' represents a bigger value than ``twenty-two''.  Once a number's value is (perhaps implicitly) represented, reasoning algorithms can then process the text, e.g., extracting the list of field goals and computing that list's maximum (first question in Figure~\ref{fig:example_1}). Learning to reason numerically over paragraphs with only question-answer supervision appears daunting for end-to-end models; our work seeks to understand \emph{if} and \emph{how} ``out-of-the-box'' neural NLP models already learn this.

We begin by analyzing the state-of-the-art NAQANet{} model \citep{drop} for DROP{}---testing it on a subset of questions that evaluate numerical reasoning (Section~\ref{sec:drop}). To our surprise, the model exhibits excellent numerical reasoning abilities. Amidst reading and comprehending natural language, the model successfully computes list maximums/minimums, extracts superlative entities (\texttt{argmax} reasoning), and compares numerical quantities. For instance, despite NAQANet{} achieving only 49 F1 on the entire validation set, it scores 89 F1 on numerical comparison questions. We also stress test the model by perturbing the validation paragraphs and find one failure mode: the model struggles to extrapolate to numbers outside its training range.

We are especially intrigued by the model's ability to learn numeracy, i.e., how does the model know the value of a number given its embedding? The model uses standard embeddings (GloVe{} and a Char-CNN) and receives no direct supervision for number magnitude/ordering. To understand how numeracy emerges, we probe token embedding methods (e.g., BERT, GloVe) using synthetic list maximum, number decoding, and addition tasks (Section~\ref{sec:numeracy}).

We find that all widely-used pre-trained embeddings, e.g.,  ELMo{}~\cite{PetersELMo2018}, \bert{}~\cite{devlin2018BERT}, and GloVe{}~\cite{pennington2014glove}, capture numeracy: number magnitude is present in the embeddings, even for numbers in the thousands. Among all embeddings, character-level methods exhibit stronger numeracy than word- and sub-word-level methods (e.g., ELMo{} excels while \bert{} struggles), and character-level models learned directly on the synthetic tasks are the strongest overall. Finally, we investigate why NAQANet{} had trouble extrapolating---was it a failure in the model or the embeddings? We repeat our probing tasks and test for model extrapolation, finding that neural models struggle to predict numbers outside the training range.
\section{Numeracy Case Study: DROP QA}\label{sec:drop}

\begin{figure}[t]
\centering
\small
\tikz\node[draw=black!40,inner sep=1pt,line width=0.3mm,rounded corners=0.1cm]{
\begin{tabular}{p{0.46\textwidth}}
\dots JaMarcus Russell completed a \setulcolor{burntorange}\ul{91-yard} touchdown pass to rookie wide receiver \answer{magenta}{Chaz Schilens}.  The Texans would respond with fullback Vonta Leach getting a {1-yard touchdown run}, yet the Raiders would answer with kicker Sebastian Janikowski getting a \setulcolor{cyan}\ul{33}-yard and a \setulcolor{cyan}\ul{21}-yard field goal.  Houston would tie the game in the second quarter with kicker Kris Brown getting a \answer{cyan}{53}-yard and a \setulcolor{cyan}\ul{24}-yard field goal. Oakland would take the lead in the third quarter with wide receiver \setulcolor{magenta}\ul{Johnnie Lee Higgins} catching a \answer{burntorange}{29-yard} touchdown pass from Russell, followed up by an 80-yard punt return for a touchdown.
\end{tabular}
};

\vspace{0.1cm}
\tikz\node[draw=black!20!cyan,inner sep=1pt,line width=0.3mm,rounded corners=0.1cm]{
\begin{tabular}{p{0.46\textwidth}}
Q: How many yards was the longest field goal? A: 53
\end{tabular}
};

\vspace{0.1cm}
\tikz\node[draw=burntorange,inner sep=1pt,line width=0.3mm,rounded corners=0.1cm]{
\begin{tabular}{p{0.46\textwidth}}
Q: How long was the shortest touchdown pass? A: 29-yard
\end{tabular}
};

\vspace{0.1cm}
\tikz\node[draw=black!20!magenta,inner sep=1pt,line width=0.3mm,rounded corners=0.1cm]{
\begin{tabular}{p{0.46\textwidth}}
Q: Who caught the longest touchdown? A: Chaz Schilens
\end{tabular}
};
\caption{Three DROP{} questions that require numerical reasoning; the state-of-the-art NAQANet{} answers every question correct. Plausible answer candidates to the questions are underlined and the model's predictions are shown in bold.}
\label{fig:example_1}
\end{figure}

This section examines the state-of-the-art model for DROP{} by investigating its accuracy on questions that require numerical reasoning.

\subsection{DROP Dataset}\label{subsec:drop}

DROP{} is a reading comprehension dataset that tests numerical reasoning operations such as counting, sorting, and addition~\cite{drop}. The dataset's input-output format is a superset of SQuAD~\cite{rajpurkar2016squad}: the answers are paragraph spans, as well as question spans, number answers (e.g., 35), and dates (e.g., 03/01/2014).  The only supervision provided is the question-answer pairs, i.e., a model must learn to reason numerically while simultaneously learning to read and comprehend.

\subsection{NAQANet{} Model}

Modeling approaches for DROP{} include both semantic parsing~\cite{krishnamurthy2017semantic} and reading comprehension~\cite{yu2018qanet} models. We focus on the latter, specifically on Numerically-augmented QANet (NAQANet{}),  the current state-of-the-art model~\cite{drop}.\footnote{Result as of May 21st, 2019.} The model's core structure closely follows QANet~\cite{yu2018qanet} except that it contains four output branches, one for each of the four answer types (passage span, question span, count answer, or addition/subtraction of numbers.)

Words and numbers are represented as the concatenation of GloVe{} embeddings and the output of a character-level CNN. The model contains no auxiliary components for representing number magnitude or performing explicit comparisons. We refer readers to \citet{yu2018qanet} and \citet{drop} for further details.

\subsection{Comparative and Superlative Questions}

\begin{table*}[t]
\centering
\small
\begin{tabular}{lll}
\toprule
\bf Question Type & \bf Example  & \bf Reasoning Required   \\ \midrule
Comparative (Binary) & Which country is a bigger exporter, Brazil or Uruguay? & Binary Comparison \\
Comparative (Non-binary) & Which player had a touchdown longer than 20 yards? & Greater Than \\
Superlative (Number) & How many yards was the shortest field goal? & List Minimum \\
Superlative (Span) & Who kicked the longest field goal? & Argmax \\
\bottomrule
\end{tabular}
\caption{We focus on DROP{} \emph{Comparative} and \emph{Superlative} questions which test NAQANet{}'s numeracy.}
\label{table:examples}
\end{table*}

We focus on questions that NAQANet{} requires numeracy to answer, namely \emph{Comparative} and \emph{Superlative} questions.\footnote{DROP{} addition, subtraction, and count questions do not require numeracy for NAQANet{}, see Appendix~\ref{appendix:qanet}.}
Comparative questions probe a model's understanding of quantities or events that are ``larger'', ``smaller'', or ``longer'' than others. Certain comparative questions ask about ``either-or'' relations (e.g., first row of Table~\ref{table:examples}), which test binary comparison. Other comparative questions require more diverse comparative reasoning, such as greater than relationships (e.g., second row of Table~\ref{table:examples}).

Superlative questions ask about the ``shortest'', ``largest'', or ``biggest'' quantity in a passage. When the answer type is a \emph{number}, superlative questions require finding the maximum or minimum of a list (e.g., third row of Table~\ref{table:examples}). When the answer type is a \emph{span}, superlative questions usually require an \texttt{argmax} operation, i.e., one must find the superlative action or quantity and then extract the associated entity (e.g., fourth row of Table~\ref{table:examples}). We filter the validation set to comparative and superlative questions by writing templates to match words in the question.

\subsection{Emergent Numeracy in NAQANet{}}

NAQANet{}'s accuracy on comparative and superlative questions is significantly higher than its average accuracy on the validation set (Table~\ref{table:breakdown}).\footnote{We have a public NAQANet{} demo available \url{https://demo.allennlp.org/reading-comprehension}.}

NAQANet{} achieves 89.0 F1 on binary (either-or) comparative questions, approximately 40 F1 points higher than the average validation question and within 7 F1 points of human test performance. The model achieves a lower, but respectable, accuracy on non-binary comparisons. These questions require multiple reasoning steps, e.g., the second question in Table~\ref{table:examples} requires (1) extracting all the touchdown distances, (2) finding the distance that is greater than twenty, and (3) selecting the player associated with the touchdown of that distance.

We divide the superlative questions into questions that have number answers
and questions with span answers according to the dataset's provided answer type.
NAQANet{} achieves nearly 70 F1 on superlative questions with number answers, i.e., it can compute list maximum and minimums. The model answers about two-thirds of superlative questions with span answers correctly (66.3 F1), i.e., it can perform \texttt{argmax} reasoning.

\begin{table}[t]
\centering \small
\begin{tabular}{lccc}
 \toprule
\textbf{Question Type} & \textbf{Count} & \textbf{EM} & \textbf{F1} \\
\midrule
Human (Test Set) & 9622 & 92.4 & 96.0 \\
\addlinespace
Full Validation & 9536 & 46.2 & 49.2 \\
\hspace{0.5cm} Number Answers & 5842 & 44.3 & 44.4 \\
\addlinespace
Comparative & 704 & 73.6 & 76.4 \\
\hspace{0.5cm} Binary (either-or) & 477 & 86.0 & 89.0 \\
\hspace{0.5cm} Non-binary & 227 & 47.6 & 49.8 \\
\addlinespace
Superlative Questions & 861 & 64.6 & 67.7 \\
\hspace{0.5cm} Number Answers & 475 & 68.8 & 69.2 \\
\hspace{0.5cm} Span Answers & 380 & 59.7 & 66.3 \\
\bottomrule
\end{tabular}
\caption{NAQANet{} achieves higher accuracy on questions that require numerical reasoning (\emph{Superlative} and \emph{Comparative}) than on standard validation questions. Human performance is reported from \citet{drop}.}
\label{table:breakdown}
\end{table}

\begin{table}[tb]
\centering
\small
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{\textbf{Stress Test Dataset}} & \multicolumn{2}{c}{\textbf{All Questions}}                & \multicolumn{2}{c}{\textbf{Superlative}}        \\
                                      & F1 & $\Delta$ &  F1 & $\Delta$ \\
\cmidrule(r){1-1}
\cmidrule(lr){2-3}
\cmidrule(lr){4-5}
Original Validation Set & 49.2 & - & 67.7 & -                 \\
\addlinespace
{Add {[}1, 20{]}} & 47.7 & -1.5 & 64.1 & -3.6                  \\
{Add {[}21, 100{]} }                    & 41.4                  & -7.8                   & 40.4                  & -27.3                  \\
\addlinespace
{Multiply {[}2, 10{]}}                  & 41.1                  & -8.1                   & 39.3                  & -28.4                  \\
{Multiply {[}11, 100{]}}                & 38.8                  & -10.4                   & 32.0                  & -35.7                  \\
\addlinespace
{Digits to Words {[}0, 20{]}}           & 45.5                  & -3.7                   & 63.8                  & -3.9                  \\
{Digits to Words {[}21, 100{]}}         & 41.9                  & -7.3                   & 46.1                  & -21.6                  \\
\bottomrule
\end{tabular}
\caption{We stress test NAQANet{}'s numeracy by manipulating the numbers in the validation paragraphs. \emph{Add} or \emph{Multiply [x, y]} indicates adding or multiplying all of the numbers in the passage by a random integer in the range [x, y]. \emph{Digits $\to$ Words [x, y]} converts all integers in the passage within the range [x, y] to their corresponding word form (e.g., ``75'' $\to$ ``seventy-five'').}
\label{table:stress_test}
\end{table}

Figure~\ref{fig:example_1} shows examples of superlative questions answered correctly by NAQANet{}. The first two questions require computing the maximum/minimum of a list: the model must recognize which digits correspond to field goals and touchdowns passes, and then extract the maximum/minimum of the correct list. The third question requires \texttt{argmax} reasoning: the model must first compute the longest touchdown pass and then find the corresponding receiver ``Chaz Schilens''.

\subsection{Stress Testing NAQANet{}'s Numeracy}\label{subsec:limit}

Just how far does the numeracy of NAQANet{} go? Here, we stress test the model by automatically modifying DROP{} validation paragraphs.

We test two phenomena: larger numbers and word-form numbers. For larger numbers,
we generate a random positive integer and multiply or add that value to the numbers in each paragraph. For word forms, we replace every digit in the paragraph with its word form (e.g., ``75'' $\to$ ``seventy-five''). Since word-form numbers are usually small in magnitude when they occur in DROP{}, we perform word replacements for integers in the range [0, 100]. We guarantee the ground-truth answer is still valid by only modifying NAQANet{}'s internal representation (Appendix~\ref{appendix:internal}).

Table~\ref{table:stress_test} shows the results for different paragraph modifications. The model exhibits a tiny degradation in performance for small magnitude changes (e.g., NAQANet{} drops 1.5 F1 overall for Add [1,20]) but severely struggles on larger changes (e.g., NAQANet{} drops 35.7 F1 on superlative questions for Multiply [11,200]).
Similar trends hold for word forms: the model exhibits small drops in accuracy when converting small numbers to words (3.9 degradation on Digits to Words [0,20]) but fails on larger magnitude word forms (21.6 F1 drop over [21,100]). These results show that NAQANet{} has a strong understanding of numeracy for numbers in the training range, but, the model can fail to extrapolate to other values.

\subsection{Whence this behavior?}
NAQANet{} exhibits numerical reasoning capabilities that exceed our expectations. What enables this behavior? Aside from reading and comprehending the passage/question, this kind of numerical reasoning requires two components: numeracy (i.e., representing numbers) and comparison algorithms (i.e., computing the maximum of a list).

Although the natural emergence of comparison algorithms is surprising, previous results show neural models are capable of learning to count and sort synthetic lists of scalar values when given explicit supervision~\cite{weiss2018counting,vinyals2016set}. NAQANet{} demonstrates that a model can learn comparison algorithms while simultaneously learning to read and comprehend, even with only question-answer supervision.

How, then, does NAQANet{} know numeracy? The source of numerical information eventually lies in the token embeddings themselves, i.e.,  the character-level convolutions and GloVe{} embeddings of the NAQANet{} model. Therefore, we can understand the source of numeracy by isolating and probing these embeddings.
\section{Probing Numeracy of Embeddings}\label{sec:numeracy}

We use synthetic numerical tasks to probe the numeracy of token embeddings.

\subsection{Probing Tasks}

We consider three synthetic tasks to evaluate numeracy (Figure~\ref{fig:probing}). Appendix~\ref{app:details} provides further details on training and evaluation.

\paragraph{List Maximum} Given a list of the embeddings for five numbers, the task is to predict the index of the maximum number. Each list consists of values of similar magnitude in order to evaluate fine-grained comparisons (see Appendix~\ref{app:details}).
As in typical span selection models~\cite{Seo2017BidirectionalAF}, an LSTM reads the list of token embeddings, and a weight matrix and softmax function assign a probability to each index using the model's hidden state. We use the negative log-likelihood of the maximum number as the loss function.

\paragraph{Decoding} The decoding task probes whether number magnitude is captured (rather than the relative ordering of numbers as in list maximum).  Given a number's embedding, the task is to regress to its value, e.g., the embedding for the string ``five'' has a target of \texttt{5.0}. We consider a linear regression model and a three-layer fully-connected network with ReLU activations. The models are trained using a mean squared error (MSE) loss.

\paragraph{Addition} The addition task requires number manipulation---given the embeddings of two numbers, the task is to predict their sum. Our model concatenates the two token embeddings and feeds the result through a three-layer fully-connected network with ReLU activations, trained using MSE loss. Unlike the decoding task, the model needs to capture number magnitude internally without direct label supervision.

\paragraph{Training and Evaluation}
We focus on a numerical \emph{interpolation} setting (we revisit extrapolation in Section~\ref{subsec:extrap}): the model is tested on values that are within the training range. We first pick a range (we vary the range in our experiments) and randomly shuffle the integers over it. We then split 80\% of the numbers into a training set and 20\% into a test set. We report the mean and standard deviation across five different random shuffles for a particular range, using the exact same shuffles across all embedding methods.

Numbers are provided as integers (``75''), single-word form (``seventy-five''), floats (``75.1''), or negatives (``-75''). We consider positive numbers less than 100 for word-form numbers to avoid multiple tokens. We report the classification accuracy for the list maximum task (5 classes), and the Root Mean Squared Error (RMSE) for decoding and addition. Note that larger ranges will naturally amplify the RMSE error.

\subsection{Embedding Methods}

We evaluate various token embedding methods.~\smallskip

\noindent \textbf{Word Vectors} We use 300-dimensional GloVe{}~\cite{pennington2014glove} and word2vec{} vectors~\cite{mikolov2018advances}. We ensure all values are in-vocabulary for word vectors.~\smallskip

\noindent \textbf{Contextualized Embeddings} We use ELMo{}~\cite{PetersELMo2018} and \bert{}~\cite{devlin2018BERT} embeddings.\footnote{Since our inputs are numbers, not natural sentences, language models may exhibit strange behavior. We experimented with extracting the context-independent feature vector immediately following the character convolutions for ELMo{} but found little difference in results.} ELMo{} uses character-level convolutions of size 1--7 with max pooling. \bert{} represents tokens via sub-word pieces; we use lowercased \bertbase{} with 30k pieces.~\smallskip

\noindent \textbf{NAQANet{} Embeddings} We extract the GloVe{} embeddings and Char-CNN from the NAQANet{} model trained on DROP{}. We also consider an ablation that removes the GloVe{} embeddings.

\noindent \textbf{Learned Embeddings} We use a character-level CNN (Char-CNN) and a character-Level LSTM (Char-LSTM). We use left character padding, which greatly improves numeracy for character-level CNNs (details in Appendix~\ref{appendix:padding}).~\smallskip

\noindent \textbf{Untrained Embeddings} We consider two untrained baselines. The first baseline is random token vectors, which trivially fail to generalize (there is no pattern between train and test numbers). These embeddings are useful for measuring the improvement of pre-trained embeddings. We also consider a randomly initialized and untrained Char-CNN and Char-LSTM.~\smallskip

\noindent \textbf{Number's Value as Embedding} The final embedding method is simple: map a number's embedding directly to its value (e.g., ``seventy-five'' embeds to [75]). We found this strategy performs poorly for large ranges; using a base-10 logarithmic scale improves performance. We report this as \emph{Value Embedding} in our results.\footnote{We suspect the failures result from the raw values being too high in magnitude and/or variance for the model. We also experimented with normalizing the values to mean 0 and variance 1; a logarithmic scale performed better.}~\smallskip

All pre-trained embeddings (all methods except the Char-CNN and Char-LSTM) are fixed during training. The probing models are trained on the synthetic tasks on top of these embeddings.

\subsection{Results: Embeddings Capture Numeracy}\label{subsec:natural}

\begin{table*}[t]
    \resizebox{1\linewidth}{!}{
    \begin{tabular}{lccccccccc}
    \toprule
    \bf \textbf{Interpolation} & \multicolumn{3}{c}{\bf List Maximum (5-classes)} & \multicolumn{3}{c}{\bf Decoding (RMSE)} & \multicolumn{3}{c}{\bf Addition (RMSE)} \\

    {\textit{Integer Range}} & [0,99] & [0,999] & [0,9999] & [0,99] & [0,999] & [0,9999] & [0,99] & [0,999] & [0,9999] \\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-4}
    \cmidrule(lr){5-7}
    \cmidrule(lr){8-10}
    Random Vectors & 0.16 & 0.23 & 0.21 & 29.86 & 292.88 & 2882.62 & 42.03 & 410.33 & 4389.39  \\
    Untrained CNN & 0.97 & 0.87 & 0.84 & 2.64 & 9.67 & 44.40 & 1.41 & 14.43 & 69.14\\
    Untrained LSTM & 0.70 & 0.66 & 0.55 & 7.61 & 46.5 & 210.34 & 5.11 & 45.69 & 510.19 \\
    Value Embedding &  \textbf{0.99} & 0.88  & 0.68 & \textbf{1.20} & 11.23 & 275.50 & \textbf{0.30} & 15.98 & 654.33 \\
    \addlinespace
    {\textit{Pre-trained}} & & & & & & \\

    Word2Vec &  0.90 &  0.78  &  0.71  & 2.34 & 18.77 & 333.47 & 0.75 & 21.23 & 210.07 \\
    GloVe &  0.90 &  0.78 &  0.72 & 2.23 & 13.77 & 174.21 & 0.80 & 16.51 & 180.31 \\
    ELMo & 0.98 &  0.88 &  0.76 & 2.35 & 13.48 & 62.20 & 0.94 & 15.50 & 45.71\\
    BERT &  0.95 &  0.62 &  0.52 & 3.21 & 29.00 & 431.78 & 4.56 & 67.81 & 454.78 \\
    \addlinespace
    {\textit{Learned}} & & & & & & \\
    Char-CNN & 0.97 &  \textbf{0.93} &  \textbf{0.88} & 2.50 & \textbf{4.92} & \textbf{11.57} & 1.19 & \textbf{7.75} & \textbf{15.09}\\
    Char-LSTM & 0.98 &  0.92 &  0.76 & 2.55 & 8.65 & 18.33 & 1.21 & 15.11 & 25.37\\
    \addlinespace
    {\textit{DROP-trained}} & & & & & & \\
    NAQANet{} &  0.91 & 0.81 &  0.72 & 2.99 & 14.19  & 62.17 & 1.11 & 11.33 & 90.01 \\
    \hspace{0.2cm} - GloVe &  0.88 &  0.90 & 0.82 & 2.87  & 5.34 & 35.39 & 1.45 & 9.91 & 60.70 \\
    \bottomrule
    \end{tabular}
    }
    \caption{{\it Interpolation with integers (e.g., ``18'').} All pre-trained embedding methods (e.g., GloVe and ELMo) surprisingly capture numeracy. The probing model is trained on a randomly shuffled 80\% of the \emph{Integer Range} and tested on the remaining 20\%. The probing model architecture and train/test splits are equivalent across all embeddings. We show the mean over 5 random shuffles (standard deviation in Appendix~\ref{appendix:additional}).}
    \label{table:interpolation_digit}
\end{table*}

We find that all pre-trained embeddings \emph{contain} fine-grained information about number magnitude and order. We first focus on integers (Table~\ref{table:interpolation_digit}).

\paragraph{Word Vectors Succeed} Both word2vec{} and GloVe{} significantly outperform the random vector baseline and are among the strongest methods overall. This is particularly surprising given the training methodology for these embeddings, e.g., a continuous bag of words objective can teach fine-grained number magnitude.

\paragraph{Character-level Methods Dominate} Models which use character-level information have a clear advantage over word-level models for encoding numbers. This is reflected in our probing results: character-level CNNs are the best architecture for capturing numeracy. For example, the NAQANet{} model without GloVe{} (only using its Char-CNN) and ELMo{} (uses a Char-CNN) are the strongest pre-trained methods, and a learned Char-CNN is the strongest method overall. The strength of the character-level convolutions seems to lie in the architectural prior---an \emph{untrained} Char-CNN is surprisingly competitive. Similar results have been shown for images~\cite{saxe2011random}: random CNNs are powerful feature extractors.

\paragraph{Sub-word Models Struggle} \bert{} struggles for large ranges (e.g., 52\% accuracy for list maximum for [0,9999]). We suspect this results from sub-word pieces being a poor method to encode digits: two numbers which are similar in value can have very different sub-word divisions.

\paragraph{A Linear Subspace Exists} For small ranges on the decoding task (e.g., [0,99]), a linear model is competitive, i.e., a linear subspace captures number magnitude (Appendix~\ref{appendix:additional}). For larger ranges (e.g., [0,999]), the linear model's performance degrades, especially for \bert{}.

\paragraph{Value Embedding Fails} The Value Embedding method fails for large ranges. This is surprising as the embedding directly provides a number's value, thus, the synthetic tasks \emph{should} be easy to solve. However, we had difficulty training models for large ranges, even when using numerous architecture variants (e.g., tiny networks with 10 hidden units and tanh activations) and hyperparameters. \citet{task2018nalu} discuss similar problems and ameliorate them using new neural architectures.

\paragraph{Words, Floats, and Negatives are Captured} Finally, we probe the embeddings on word-form numbers, floats, and negatives. We observe similar trends for these inputs as integers: pre-trained models exhibit natural numeracy and learned embeddings are strong (Tables~\ref{table:floats}, \ref{table:negative}, and \ref{table:interpolation_words}). The ordering of the different embedding methods according to performance is also relatively consistent across the different input types. One notable exception is that \bert{} struggles on floats, which is likely a result of its sub-word pieces. We do not test word2vec{} and GloVe{} on floats/negatives because they are out-of-vocabulary.

\begin{table}[t]
    \centering
    \begin{tabular}{lcc}
    \toprule
    \bf  \textbf{Interpolation} & \multicolumn{2}{c}{\bf List Maximum (5-classes)}\\
    {\textit{Float Range}} & [0.0,99.9] & [0.0,999.9] \\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-3}
    Rand. Vectors & 0.18 $\pm$ 0.03 & 0.21 $\pm$ 0.04 \\
    ELMo & 0.91 $\pm$ 0.03 & 0.59 $\pm$ 0.01 \\
    BERT & 0.82 $\pm$ 0.05 & 0.51 $\pm$ 0.04\\
    Char-CNN & 0.87 $\pm$ 0.04 & 0.75 $\pm$ 0.03 \\
    Char-LSTM & 0.81 $\pm$ 0.05 & 0.69 $\pm$ 0.02 \\
    \bottomrule
    \end{tabular}
    \caption{{\it Interpolation with floats (e.g., ``18.1'') for list maximum.} Pre-trained embeddings capture numeracy for float values. The probing model is trained on a randomly shuffled 80\% of the \emph{Float Range} and tested on the remaining 20\%. See the text for details on selecting decimal values. We show the mean alongside the standard deviation over 5 different random shuffles.}
    \label{table:floats}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{lc}
    \toprule
    \bf  \textbf{Interpolation} & \multicolumn{1}{c}{\bf List Maximum (5-classes)}\\
    {\textit{Integer Range}} & [-50,50]\\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-2}
    Rand. Vectors & 0.23 $\pm$ 0.12 \\
    Word2Vec & 0.89 $\pm$ 0.02\\
    GloVe & 0.89 $\pm$ 0.03 \\
    ELMo & 0.96 $\pm$ 0.01\\
    BERT & 0.94 $\pm$ 0.02\\
    Char-CNN & 0.95 $\pm$ 0.07 \\
    Char-LSTM & 0.97 $\pm$ 0.02 \\
    \bottomrule
    \end{tabular}
    \caption{{\it Interpolation with negatives (e.g., ``-18'') on list maximum.} Pre-trained embeddings capture numeracy for negative values.}
    \label{table:negative}
\end{table}

\subsection{Probing Models Struggle to Extrapolate}\label{subsec:extrap}

Thus far, our synthetic experiments evaluate on held-out values \emph{within} the same range as the training data (i.e., numerical interpolation). In Section~\ref{subsec:limit}, we found that NAQANet{} struggles to extrapolate to values \emph{outside} the training range. Is this an idiosyncrasy of NAQANet{} or is it a more general problem? We investigate this using a numerical extrapolation setting: we train models on a specific integer range and test them on values greater than the largest training number and smaller than the smallest training number.

\setlength{\tabcolsep}{3pt}
\begin{table}[t]
    \centering
    \small
    \begin{tabular}{lccc}
    \toprule
    \bf  \textbf{Extrapolation} & \multicolumn{3}{c}{\bf List Maximum (5-classes)}\\
    {\textit{Test Range}} & [151,160] & [151,180] & [151,200]\\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-4}
    Rand. Vectors & 0.17 & 0.22 & 0.15 \\
    Untrained CNN & 0.80 & 0.47 & 0.41\\
    \addlinespace
    {\textit{Pre-trained}} & & &\\
    Word2Vec & 0.14 & 0.16 & 0.11 \\
    GloVe & 0.19 & 0.17 & 0.21 \\
    ELMo & 0.65 & 0.57 & 0.38\\
    BERT & 0.35 & 0.11 & 0.14\\
    \addlinespace
    {\textit{Learned}} & & &\\
    Char-CNN & 0.81 & 0.75 & 0.73\\
    Char-LSTM & \textbf{0.88} & \textbf{0.84} & \textbf{0.82}\\
    \addlinespace
    {\textit{DROP}} & & & \\
    NAQANet{} & 0.31 & 0.29 & 0.25\\
    \hspace{0.2cm} - GloVe & 0.58 & 0.53 & 0.48\\
    \bottomrule
    \end{tabular}
    \caption{{\it Extrapolation on list maximum.} The probing model is trained on the integer range [0,150] and evaluated on integers from the \emph{Test Range}. The probing model struggles to extrapolate when trained on the pre-trained embeddings.}
    \label{table:extrapolation_digits}
\end{table}

\paragraph{Extrapolation for Decoding and Addition} For decoding and addition, models struggle to extrapolate. Figure~\ref{fig:plot} shows the predictions for models trained on 80\% of the values from [-500,500] and tested on held-out numbers in the range [-2000, 2000] for six embedding types. The embedding methods fail to extrapolate in different ways, e.g., predictions using word2vec{} decrease almost monotonically as the input increases, while predictions using \bert{} are usually near the highest training value. \citet{task2018nalu} also observe that models struggle outside the training range; they attribute this to failures in neural models themselves.

\paragraph{Extrapolation for List Maximum} For the list maximum task, accuracies are closer to those in the interpolation setting, however, they still fall short. Table~\ref{table:extrapolation_digits} shows the accuracy for models trained on the integer range [0,150] and tested on the ranges [151,160], [151,180], and [151,200]; all methods struggle, especially token vectors.

\paragraph{Augmenting Data to Aid Extrapolation} Of course, in many real-word tasks it is possible to ameliorate these extrapolation failures by augmenting the training data (i.e., turn extrapolation into interpolation). Here, we apply this idea to aid in training NAQANet{} for DROP{}. For each superlative and comparative example, we duplicate the example and modify the numbers in its paragraph using the \emph{Add} and \emph{Multiply} techniques described in Section \ref{subsec:limit}. Table \ref{table:count} shows that this data augmentation can improve both interpolation and extrapolation, e.g., the accuracy on superlative questions with large numbers can \emph{double}.
\section{Discussion and Related Work}\label{sec:related}

An open question is how the training process elicits numeracy for word vectors and contextualized embeddings. Understanding this, perhaps by tracing numeracy back to the training data, is a fruitful direction to explore further (c.f., influence functions~\cite{koh2017influence,brunet2019understanding}).

More generally, numeracy is one type of emergent knowledge. For instance, embeddings may capture the size of objects~\cite{forbes2017verb}, speed of vehicles, and many other ``commonsense'' phenomena~\cite{yang2018common}. \citet{vendrov2015order} introduce methods to encode the order of such phenomena into embeddings for concepts such as hypernymy; our work and \citet{yang2018common} show that a relative ordering naturally emerges for certain concepts.

In concurrent work, \citet{naik2019numeracy} also explore numeracy in word vectors. Their methodology is based on variants of nearest neighbors and cosine distance; we use neural network probing classifiers which can capture highly non-linear dependencies between embeddings. We also explore more powerful embedding methods such as ELMo, BERT, and learned embedding methods.

\paragraph{Probing Models} Our probes of \emph{numeracy} parallel work in understanding the linguistic capabilities (\emph{literacy}) of neural models~\cite{conneau2018cram,liu2019linguistic}. LSTMs can remember sentence length, word order, and which words were present in a sentence~\cite{adi2017lstm}. \citet{khandelwal2018lm} show how language models leverage context, while \citet{linzen2016assessing} demonstrate that language models understand subject-verb agreement.

\paragraph{Numerical Value Prediction} \citet{Spithourakis2018numeracy} improve the ability of language models to \emph{predict} numbers, i.e., they go beyond categorical predictions over a fixed-size vocabulary. They focus on improving models; our focus is probing embeddings. \citet{kotnis2019learning} predict numerical attributes in knowledge bases, e.g., they develop models that try to predict the population of Paris.

\paragraph{Synthetic Numerical Tasks} Similar to our synthetic numerical reasoning tasks, other work considers sorting~\cite{graves2014neural}, counting~\cite{weiss2018counting}, or decoding tasks~\cite{task2018nalu}. They use synthetic tasks as a testbed to prove or design better models, whereas we use synthetic tasks as a probe to understand token embeddings. In developing the Neural Arithmetic Logic Unit, \citet{task2018nalu} arrive at similar conclusions regarding extrapolation: neural models have difficulty outputting numerical values outside the training range.
\section{Conclusion}\label{sec:conclusion}

How much do NLP models know about numbers? By digging into a surprisingly successful model on a numerical reasoning dataset (DROP), we discover that pre-trained token representations naturally encode numeracy.

We analyze the limits of this numeracy, finding that CNNs are a particularly good prior (and likely the cause of ELMo's superior numeracy compared to BERT) and that it is difficult for neural models to extrapolate beyond the values seen during training. There are still many fruitful areas for future research, including discovering \emph{why} numeracy naturally emerges in embeddings, and what other properties are similarly emergent.
\section{NAQANet{} Details and Numeracy Requirements}\label{appendix:qanet}

Aside from typical paragraph spans, NAQANet{} can also output spans in the question, counts from 0--9, and arithmetic expressions. The arithmetic expressions are generated by first extracting all the numbers from the passage and then assigning a coefficient in \{-1,0,1\} to each number. The final answer is calculated as the sum of each number using its associated coefficient.

Addition and subtraction questions are present in DROP{} but do not require numeracy for NAQANet{}: the model can output the correct coefficients based on the surrounding context words without understanding number magnitude. For example, consider a paragraph containing,  ``Superbowl XXXI occurred in 1997 and Superbowl XXXVII occurred in 2003'', and a question ``How many years after Superbowl XXXI did Superbowl XXXVII occur?''. The model can output coefficient +1 on ``2003'' and -1 on ``1997'' and answer correctly without understanding the magnitude of the two years. Counting questions also do not require reading or manipulating numbers in text: the model only needs to output the correct 10-way classification value.

\section{Left-padding for Enhancing Char-CNN Numeracy}\label{appendix:padding}

In modern deep learning and NLP frameworks, pad tokens are often appended to the right of inputs to ensure equal lengths inside each batch (for efficiency). When using \emph{character}-level convolutions, padding must be added to the characters of a word when its length is smaller than the minimum kernel width (for correctness). Naturally, the right padding implementation is re-used, e.g., the two numbers ``11'' and ``110'' are represented as:
\begin{align}
\begin{array}{rcl}
\nonumber
 11 &=& 11\oldemptyset\oldemptyset\oldemptyset\\
 110 &=& 110\oldemptyset\oldemptyset,
\end{array}
\label{eq:mape}
\end{align}
where $\oldemptyset$ is the pad token. However, right padding is detrimental to learning numeracy: the ``hundreds'' and ``tens'' digits are now in the same position in the two tokens. We found that left padding numbers substantially decreases the generalization error and convergence rate for Char-CNN models trained on the synthetic tasks.\footnote{Padding on the left and right via {\tt SAME} convolutions also mitigates this issue.}

\section{Training Details for Probing}\label{app:details}

We create training/test splits for the addition task in the following manner. We first shuffle and split an integer range, putting 80\% into train and 20\% into test. Next, we enumerate all possible pairs of numbers in the two respective splits. When using large ranges such as [0,999], we sub-sample a random 10\% of the training and test pairs.

For the list maximum task, we first shuffle and split the data, putting 80\% into a training pool of numbers and 20\% into a test pool. In initial experiments, we created the lists of five numbers by sampling uniformly over the training/test pool. However, as the random samples will likely be spread out over the range, the numbers are easy to distinguish. We instead create 100,000 training examples and 10,000 examples in the following manner. We first sample a random integer from the training or test pool. Next, we sample from a Gaussian with mean zero and variance equal to 0.01 times the total size of the range. Finally, we add the random Gaussian sample to the random integer, and round to the nearest value in the pool. This forces the numbers to be nearby.

\section{Additional Synthetic Results}\label{appendix:additional}

\subsection{Linear Regression Accuracies}

Table~\ref{table:linear_regression} provides accuracies for number decoding using linear regression (interpolation setting).

\begin{table}[h!]
    \centering
    \begin{tabular}{lccc}
    \toprule
    \bf Interpolation & \multicolumn{3}{c}{\bf Decoding (RMSE)}  \\
    {\textit{Integer Range}} & [0,50] & [-50,50] & [0,999] \\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-4}
    Random & 13.86 & 29.46 & 275.41 \\
    Word2Vec & 4.15 & 8.93 & 29.04\\
    GloVe & 3.21 & 5.76 & 23.27\\
    ELMo & 1.20 & 2.89 & 21.53 \\
    BERT & 3.23 & 7.86 & 64.42\\
    \bottomrule
    \end{tabular}
    \caption{{\it Number Decoding interpolation accuracy with linear regression.} Linear regression is competitive to the fully connected probe for smaller numbers.}
    \label{table:linear_regression}
\end{table}

\subsection{Variances on Digit Form Results}

Table~\ref{table:interpolation_digit_variance} shows the mean and standard deviation for the synthetic tasks using five random shuffles.

\newpage
\subsection{Float Values}

We test floats with one decimal point. We follow the setup for the list maximum task (Appendix~\ref{app:details}) with a minor modification. For 50\% of the training/test lists, we reuse the same integer five times but sample a different random value to put after the decimal point. For example, 50\% of the lists are of the form: [15.3, 15.6, 15.1, 15.8, 15.2] (the same base integer is repeated with a different decimal), and 50\% are random integers with a random digit after the decimal: [11.7, 16.4, 9.3, 7.9, 13.3]. This forces the model to consider the numbers on both the left and the right of the decimal.

\subsection{Word Form Results}

Table~\ref{table:interpolation_words} presents the results using word-forms. We do not use numbers larger than 100 as they consist of multiple words.

\section{Automatically Modifying DROP Paragraphs}\label{appendix:internal}

Modifying a DROP{} paragraph automatically is challenging as it may change the answer to the question if done incorrectly. We also cannot modify the answer because many DROP{} questions have count answers. To guarantee that the original annotated answer can still be used, we perform the number transformation in the following manner. We keep the original passage text the same, but, we modify the model's internal embeddings for the numbers directly. In other words, the model uses exactly the same embeddings for the original text except for the modified numbers. The model then needs to find the correct index (e.g. the index of the correct span, or the index of the correct number) given these modified embeddings.

\begin{table*}[t]
    \resizebox{1\linewidth}{!}{
    \begin{tabular}{lccccccccc}
    \toprule
    \bf \textbf{Interpolation} & \multicolumn{3}{c}{\bf List Maximum (5-classes)} & \multicolumn{3}{c}{\bf Decoding (RMSE)}  & \multicolumn{3}{c}{\bf Addition (RMSE)} \\

    {\textit{Integer Range}} & [0,99] & [0,999] & [0,9999] & [0,99] & [0,999] & [0,9999]  & [0,99] & [0,999] & [0,9999] \\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-4}
    \cmidrule(lr){5-7}
    \cmidrule(lr){8-10}
    Random Vectors & 0.16 $\pm$ 0.03 &  0.23 $\pm$ 0.12 & 0.21 $\pm$ 0.02 & 29.86 $\pm$ 4.44 & 292.88 $\pm$ 13.48 & 2882.62 $\pm$ 71.68 & 42.03 $\pm$ 7.79 & 410.33 $\pm$ 16.05 & 4389.39 $\pm$ 310.91 \\
    Untrained CNN & 0.97 $\pm$ 0.01 & 0.87 $\pm$ 0.02 &  0.84 $\pm$ 0.03 & 2.64 $\pm$ 0.68 & 9.67 $\pm$ 1.17 & 44.40 $\pm$ 4.98 & 1.41 $\pm$ 0.05 & 14.43 $\pm$ 1.90 & 69.14 $\pm$ 21.54 \\
    Untrained LSTM & 0.70 $\pm$ 0.05 & 0.66 $\pm$ 0.03 & 0.55 $\pm$ 0.02 & 7.61 $\pm$ 1.33 & 46.5 $\pm$ 5.65 & 210.34 $\pm$ 9.91 & 5.11 $\pm$ 2.1 & 45.69 $\pm$ 3.78 & 510.19 $\pm$ 31.45 \\
    Value Embedding &  0.99 $\pm$ 0.01 & 0.88 $\pm$ 0.04 & 0.68 $\pm$ 0.06 & 1.20 $\pm$ 0.76  & 11.23 $\pm$ 1.35 & 275.5 $\pm$ 135.59 & 0.30 $\pm$ 0.01 & 15.98 $\pm$ 3.62 & 654.33 $\pm$ 69.91 \\
    \addlinespace
    {\textit{Pre-trained}} & & & & & & \\

    Word2Vec &  0.90 $\pm$ 0.03 &  0.78 $\pm$ 0.05 &  0.71 $\pm$ 0.03  & 2.34 $\pm$ 1.44 & 18.77 $\pm$ 4.40 & 333.47 $\pm$ 14.83 & 0.75 $\pm$ 0.41 & 21.23 $\pm$ 3.53 & 210.07 $\pm$ 30.56 \\
    GloVe &  0.90 $\pm$ 0.02 &  0.78 $\pm$ 0.04 &  0.72 $\pm$ 0.02 & 2.23 $\pm$ 1.26 & 13.77 $\pm$ 3.23 & 174.21 $\pm$ 31.91 & 0.80 $\pm$ 0.30 & 16.51 $\pm$ 1.17 & 180.31 $\pm$ 18.97 \\
    ELMo & 0.98 $\pm$ 0.04 &  0.88 $\pm$ 0.02 &  0.76 $\pm$ 0.04 & 2.35 $\pm$ 0.65 & 13.48 $\pm$ 2.19 & 62.2 $\pm$ 5.04 & 0.94 $\pm$ 0.35 & 15.50 $\pm$ 2.49 & 45.71 $\pm$ 18.69  \\
    BERT &  0.95 $\pm$ 0.02 &  0.62 $\pm$ 0.01 &  0.52 $\pm$ 0.07 & 3.21 $\pm$ 0.31 & 29.00 $\pm$ 7.93 & 431.78 $\pm$ 30.27 & 4.56 $\pm$ 1.76 & 67.81 $\pm$ 30.34 & 454.78 $\pm$ 91.07\\
    \addlinespace
    {\textit{Learned}} & & & & & & \\
    Char-CNN & 0.97 $\pm$ 0.02 &  0.93 $\pm$ 0.01 & 0.88 $\pm$ 0.02 & 2.50 $\pm$ 0.49 & 4.92 $\pm$ 0.97 & 11.57 $\pm$ 1.34 & 1.19 $\pm$ 0.18 & 7.75 $\pm$ 1.85 & 15.09 $\pm$ 0.90 \\
    Char-LSTM & 0.98 $\pm$ 0.02 &  0.92 $\pm$ 0.02 &  0.76 $\pm$ 0.03 & 2.55 $\pm$ 1.81 & 8.65 $\pm$ 1.11 & 18.33 $\pm$ 3.21 & 1.21 $\pm$ 0.10 & 15.11 $\pm$ 1.87 & 25.37 $\pm$ 3.40 \\
    \addlinespace
    {\textit{DROP-trained}} & & & & & & \\
    NAQANet{} &  0.91 $\pm$ 0.01 & 0.81 $\pm$ 0.02 &  0.72 $\pm$ 0.03 & 2.99 $\pm$ 1.11 & 14.19 $\pm$ 3.75 & 62.17 $\pm$ 4.31 & 1.11 $\pm$ 0.41 & 11.33 $\pm$ 1.67 & 90.01 $\pm$ 17.88 \\
    \hspace{0.2cm} - GloVe &  0.88 $\pm$ 0.02 & 0.90 $\pm$ 0.03 & 0.82 $\pm$ 0.02 &
    2.87 $\pm$ 0.83 & 5.34 $\pm$ 0.77 & 35.39 $\pm$ 4.32 & 1.45 $\pm$ 0.95 & 9.91 $\pm$ 1.45 & 60.70 $\pm$ 13.04 \\
    \bottomrule
    \end{tabular}
    }
    \caption{Mean and standard deviation for Table~\ref{table:interpolation_digit} (interpolation tasks with integers).}
    \label{table:interpolation_digit_variance}
\end{table*}

\begin{table*}[t]
    \resizebox{1\linewidth}{!}{
    \begin{tabular}{lcccccc}
    \toprule
    \bf \textbf{Interpolation} &
    \multicolumn{2}{c}{\bf List Maximum (5-classes)}
    & \multicolumn{2}{c}{\bf Decoding (MSE)} & \multicolumn{2}{c}{\bf Addition (MSE)}  \\
    {\textit{Integer Range}} & [zero, fifty] & [zero,ninety-nine] & [zero, fifty] & [zero,ninety-nine] & [zero, fifty] & [zero,ninety-nine] \\
    \cmidrule(r){1-1}
    \cmidrule(lr){2-3}
    \cmidrule(lr){4-5}
    \cmidrule(lr){6-7}
    Random Vectors & 0.16 $\pm$ 0.03 & 0.23 $\pm$ 0.12 & 15.43 $\pm$ 1.13  & 29.86 $\pm$ 4.44 & 23.14 $\pm$ 4.46 & 43.86 $\pm$ 1.41 \\
    \addlinespace
    {\textit{Pre-trained}} & & & & & & \\
    Word2Vec & 0.96 $\pm$ 0.05 & 0.91 $\pm$ 0.05 & 2.40 $\pm$ 0.64 & 3.94 $\pm$ 1.88 & 2.10 $\pm$ 0.31 & 4.10 $\pm$ 0.98 \\
    GloVe  & 0.93 $\pm$ 0.07 & 0.90 $\pm$ 0.03 & 3.02 $\pm$ 0.98 & 4.75 $\pm$ 1.59 & 3.33 $\pm$ 0.51 & 4.81 $\pm$ 1.00 \\
    ELMo & 0.75 $\pm$ 0.04 & 0.82 $\pm$ 0.10 & 3.86 $\pm$ 0.92 & 7.31 $\pm$ 2.06 & 3.91 $\pm$ 0.55 & 6.97 $\pm$ 1.19  \\
    BERT & 0.61 $\pm$ 0.11 & 0.66 $\pm$ 0.05 & 7.59 $\pm$ 1.37 & 13.55 $\pm$ 3.15 & 7.33 $\pm$ 1.05 & 12.91 $\pm$ 2.57 \\
    \addlinespace
    {\textit{Learned}} & & & & & & \\
    Char-CNN & 0.98 $\pm$ 0.01 & 0.99 $\pm$ 0.01 & 5.73 $\pm$ 0.93 & 7.85 $\pm$ 2.36 & 6.11 $\pm$ 0.55 & 7.11 $\pm$ 2.34 \\
    Char-LSTM & 0.89 $\pm$ 0.06 & 0.86 $\pm$ 0.08 & 8.45 $\pm$ 4.97 & 7.31 $\pm$ 2.06 & 8.91 $\pm$ 3.11 & 9.51 $\pm$ 1.19 \\
    \addlinespace
    {\textit{DROP-trained}} & & & & & & \\
    NAQANet{} & 0.98 $\pm$ 0.01 & 0.97 $\pm$ 0.02 & 3.17 $\pm$ 1.05 & 4.31 $\pm$ 0.68 & 3.05 $\pm$ 0.87 & 3.95 $\pm$ 0.33 \\
    \hspace{0.5cm} - GloVe & 0.96 $\pm$ 0.03 & 0.97 $\pm$ 0.02 & 7.81 $\pm$ 1.34 & 9.43 $\pm$ 2.31 & 8.55 $\pm$ 0.88 & 10.01 $\pm$ 2.07\\
    \bottomrule
    \end{tabular}
    }
    \caption{{\it Interpolation task accuracies with word form (e.g., ``twenty-five'').} The model is trained on a randomly shuffled 80\% of the \emph{Integer Range} and tested on the remaining 20\%. We show the mean and standard deviation for five random shuffles.}
    \label{table:interpolation_words}
\end{table*}

\section{Extrapolation with Data Augmentation}

For each superlative and comparative example, we modify the numbers in its paragraph using the \emph{Add} and \emph{Multiply} techniques mentioned in Section \ref{subsec:limit}. We first multiply the paragraph's numbers by a random integer from [1, 10], and then add another random integer from [0, 20]. We train NAQANet{} on the original paragraph and an additional modified version for all training examples. We use a single additional paragraph for computational efficiency; augmenting the data with more modified paragraphs may further improve results.

We test NAQANet{} on the original validation set, as well as a \emph{Bigger} validation set. We created the Bigger validation set  by multiplying each paragraph's numbers by a random integer from [11,20] and then adding a random value from [21,40]. Note that this range is larger than the one used for data augmentation.

Table \ref{table:count} shows the results of NAQANet{} trained with data augmentation. Data augmentation provide small gains on the original superlative and comparative question subset, and significant improvements on the \emph{Bigger} version (it doubles the model's F1 score for superlative questions).

\begin{table*}[t]
    \centering \small
    \begin{tabular}{lcccccc}
    \toprule
    & \multicolumn{2}{c}{\textbf{Superlative}} & \multicolumn{2}{c}{\textbf{Comparative}} & \multicolumn{2}{c}{\textbf{All Validation}} \\
    & Original & Bigger & Original & Bigger & Original & Bigger \\

    \cmidrule(lr){2-3}
    \cmidrule(lr){4-5}
    \cmidrule(lr){6-7}
    NAQANet & 64.5 / 67.7 &  30.0 / 32.2 & 73.6 / 76.4 & 70.3 / 73.0 &  \textbf{46.2} / 49.2 & 38.7 / 41.4  \\

    + Data Augmentation & \textbf{67.6} / \textbf{70.9} & \textbf{59.2} / \textbf{62.4} & \textbf{76.0} / \textbf{77.7} & \textbf{75.0} / \textbf{76.8} & 46.1 / \textbf{49.3} & \textbf{42.8} /  \textbf{45.8} \\
    \bottomrule
    \end{tabular}
    \caption{Data augmentation improves NAQANet{}'s interpolation and extrapolation results. We created the \emph{Bigger} version of DROP{} by multiplying numbers in the passage by a random integer from [11, 20] and then adding a random integer from [21, 40]. Scores are shown in EM / F1 format.}
    \label{table:count}
\end{table*}

\section*{Acknowledgements}

We thank Mark Neumann, Suchin Gururangan, Pranav Goel,
Shi Feng, Nikhil Kandpal, Dheeru Dua, the members of AllenNLP and UCI NLP, and
the reviewers for their valuable feedback.

redpajama_set_name: RedPajamaArXiv


Set against the backdrop of 19th-century France, Les Miserables tells an enthralling story of broken dreams and unrequited love, passion, sacrifice and redemption-a timeless testament to the survival of the human spirit. Jackman plays ex-prisoner Jean Valjean, hunted for decades by the ruthless policeman Javert (Crowe) after he breaks parole. When Valjean agrees to care for factory worker Fantine's (Hathaway) young daughter, Cosette, their lives change forever.


Cardholder Present payment acceptance is made more reliable and more secure for your customers with a range of TRANSAXpay processing solutions from FIS. In addition, our solutions are designed to reduce the scope and cost of merchant assessment and to help you achieve the requirements that are required of you by the Payment Card Industry Data Security Standards (PCI DSS).
The FIS Payment Gateway for fast card authorisation forwarding, accurate settlement file submission, secure data hosting and instant financial reporting. PCI Level 1 certified.
Card data encryption within the PIN Entry Device (PED) – with the TRANSAXpay Point-to-Point Encryption (P2PE), a PCI accredited application.
Our payment processing and encryption solutions are fully compatible with Contactless payments. We can provide the right advice for hardware provision, service and maintenance that also extends to sourcing and purchase / leasing options.


text:
Q: Sporadic "Method 'get_Session' in type from assembly does not have an implementation" Suddenly the web app that I develop started to give this error message - to the user, but not to me, and only sometimes.
I know that this error can be caused by interface assembly and implementation assembly reference versions mismatch. But I did not update Sharp's version for a long time (still use very old one for this project). Also, the error does not happen always, if it was wrong assemblies I suppose it would always fail.
What can be the reason? Are there any tracing/loggin tools in framework to find out?
Method 'get_Session' in type 'Orders.Data.SafeSessionStorage'
from assembly 'Orders.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'
does not have an implementation."

System.TypeLoadException: Method 'get_Session' in type 'Orders.Data.SafeSessionStorage' from assembly 'Orders.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.
   at Orders.Web.MvcApplication.InitializeNHibernateSession()
   at Orders.Web.MvcApplication.<Application_BeginRequest>b__1d()
   at SharpArch.Data.NHibernate.NHibernateInitializer.InitializeNHibernateOnce(Action initMethod)
   at Orders.Web.MvcApplication.Application_BeginRequest(Object sender, EventArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Here's the SafeSessionStorage. It is a slightly modified version of SharpArch's one, to support running in background threads.
public class SafeSessionStorage : ISessionStorage
{
  [ThreadStatic]
  private static ISession _session;

  public ISession Session
  {
     get
     {
        HttpContext context = HttpContext.Current;
        if (context == null)
           return _session;
        else
        {
           ISession session = context.Items[factoryKey] as ISession;
           return session;
        }
     }
     set
     {
        HttpContext context = HttpContext.Current;
        if (context == null)
           _session = value;
        else
           context.Items[factoryKey] = value;
     }
  }

  public string FactoryKey
  {
     get { return factoryKey; }
  }

  public static void End()
  {
     if (_session != null)
        _session.Close();
     _session = null;
  }

  public void EndRequest()
  {
     ISession session = Session;

     if (session != null)
     {
        session.Close();
        HttpContext context = HttpContext.Current;
        if (context != null)
           context.Items.Remove(factoryKey);
        else
           _session = null;
     }
  }

  private string factoryKey = NHibernateSession.DefaultFactoryKey;
}

Here's where error happens:
  private void InitializeNHibernateSession()
  {
     NHibernateInitHelper.InitSession(safeSessionStorage,
        Server.MapPath("~/NHibernate.config"),
        Server.MapPath("~/bin/Orders.Data.dll"));
  }

Here InitSession expects ISessionStorage and is passed SafeSessionStorage, so I suppose that's where type checking fails. And I would suspect assemblies versions but, as I said, it always works for me and sometimes works for the user.

A: I would better accept sehe's comment as answer, but anyway. The problem was a StackOverflowException because of recursive DB data. To debug this I had to add logging to many lines inside the suspected code (the error happened on SOAP service acccess and mapping data using DB), and then analyze. Another approach was to deploy debug build (this is dev server) and use WinDbg, which gave correct exception code 0xe053534f so that I could concentrate on code that may cause this (recursive LINQ method to collect related products).
The low drive space was a side effect of DW20.exe (dr. watson) eating 1.5 GB space (and CPU).
Event viewer was a little help because it showed the too generic "kernel32.dll, address 0x0000bee7" error that could be stack overflow and could be anything else.
Getting "method does not have implementation" is the last information I'd expect from such conditions.

redpajama_set_name: RedPajamaStackExchange


Our Vision: Our vision is to be an internationally top-ranked, research-driven, independent management institution that grooms future leaders for India and the world.
- To create and disseminate research-based knowledge in management that influences scholarship, practice and policy.
- To develop and enhance business leadership through innovative world-class programmes in management.
- To utilise the knowledge and expertise to engage with business, government and society, and to contribute to the welfare and development of the community at the local, national, and global levels.
- To have distinguished faculty from India and abroad teach courses and provide a global perspective in Post Graduate and Executive Education programmes.


text:
Small Cap Value Fund - SBRVX
The Segall Bryant & Hamill Small Cap Value Fund (the "Fund") seeks long term capital appreciation.
Small Cap Value Fund 7.64 7.30 24.62 24.62 9.75 8.71 9.38
Russell 2000 Value 8.49 7.87 22.39 22.39 4.77 6.99 7.75
Small Cap Value Fund 4.08 7.64 7.30 24.62 24.62 9.75 8.71 9.38
Russell 2000 Value 3.50 8.49 7.87 22.39 22.39 4.77 6.99 7.75
Small Cap Value Fund 12.80 -0.23 0.71
* Since Inception Date: 12/9/2019
Performance data quoted represents past performance and does not guarantee future results. Performance information for the retail class shares prior to their inception is based on the performance of the institutional class. Current performance may be lower or higher than the performance quoted. To obtain current performance as of the most recent month-end, please call (800) 392-2673. Average annual total returns reflect the reinvestment of dividends, capital gains distributions, all fee waivers and expense reimbursements. If imposed, the fee would reduce the performance quoted. Investment return and principal value will vary, and shares, when redeemed, may be worth more or less than their original cost.
From December 9, 2019 until at least December 31, 2021, the Adviser has contractually agreed to waive the investment advisory and/or administration fees and/or to reimburse other expenses (not including acquired fund fees and expenses, taxes, brokerage expenses, and extraordinary expenses), so that the ratio of expenses of average net assets as reported in the Fund's Financial Highlights will be no more than 1.14% and 0.99% to the Fund's Retail Class and Institutional Class, respectively, for such period. This agreement may not be terminated or modified by the Adviser prior to December 31, 2021 without the approval of the Board of Trustees.
Mark T. Dickherber, CFA, CPA
Principal, Director of Small Cap Strategies
Mark Dickherber is a Principal and Director of Segall Bryant & Hamill's Small Cap strategies. He is the lead Portfolio Manager for our Small Cap Value strategy and the Co-Portfolio Manager for our Small Cap Core and Small Cap Value Select strategies. He is also responsible for equity research related to healthcare, utilities and REITs within the respective portfolios. Prior to joining SBH, Mr. Dickherber served as Director of Research for Kennedy Capital Management, where he had worked since 1996. Mr. Dickherber graduated magna cum laude from the University of Missouri-St. Louis with a B.S. in Accounting. He served as Treasurer for the CFA Society of St. Louis from 2004-2006 and as a Director from 2006-2008. Mr. Dickherber has been in the investment industry since 1996 and has earned Chartered Financial Analyst (CFA) and Certified Public Accountant (CPA) designations.
Shaun P. Nicholson
Principal, Senior Portfolio Manager
Shaun Nicholson is a Principal and Senior Portfolio Manager for Segall Bryant & Hamill's Small Cap strategies. He is the lead Portfolio Manager for our Small Cap Value Concentrated strategy and the Co-Portfolio Manager for our Small Cap Value strategy. Mr. Nicholson is also responsible for research related to materials, autos/transports, industrials and regional banks within the respective portfolios. He joined SBH in 2011 after having spent more than six years at Kennedy Capital Management. Prior to Kennedy, Mr. Nicholson's experience was as an Associate Portfolio Manager at U.S. Bancorp Asset Management and as a Financial Analyst at The Boeing Company. Mr. Nicholson earned a B.S. from Seton Hall University and earned an MBA from the University of Missouri-St. Louis. He has been in the investment industry since 2002.
Jeffrey C. Paulis, CFA
Jeff Paulis is a Principal and Senior Portfolio Manager for Segall Bryant & Hamill's Small Cap strategies. He is the lead Portfolio Manager for our SMID Cap and Small Cap Core strategies. Mr. Paulis is also responsible for equity research within the industrials sector for the respective portfolios. Prior to joining SBH in 2003, his professional experience included serving as a Research Analyst at McDonald Investments. Prior to joining McDonald Investments, he served as an intern at Kennedy Capital Management. He is a graduate of Saint Louis University with a B.S. in Business Administration and holds an MBA from the University of Chicago's Booth School of Business. Mr. Paulis has been in the investment industry since 2001 and has earned the Chartered Financial Analyst (CFA) designation.
Zachary T. Rosenstock, CFA
Zach Rosenstock is a Senior Equity Analyst for Segall Bryant & Hamill's Small Cap Equity strategies. Mr. Rosenstock is responsible for technology related research within the respective strategies. His professional experience includes serving as a Research Analyst at Great Lakes Advisors. He is a graduate of Boston College with a B.A. in Economics. Mr. Rosenstock has been in the investment industry since 2005 and has earned the Chartered Financial Analyst (CFA) designation.
Eric D. Hines, CFA
Eric Hines is an Equity Analyst for Segall Bryant & Hamill's Small Cap Equity strategies. Mr. Hines is responsible for equity research related to energy, consumer staples and consumer discretionary within the respective portfolios. His professional experience includes serving as both an Account Executive and Consultant at FactSet Research Systems. Prior to joining FactSet, he served as an intern at Kennedy Capital Management. He is a graduate of Saint Louis University with a B.S. in Business Administration. Mr. Hines has been in the investment industry since 2007 and has earned the Chartered Financial Analyst (CFA) designation.
Account Access Month-End Holdings Distributions

redpajama_set_name: RedPajamaCommonCrawl


text:
Jeune :
 jeûne, privation, volontaire ou non, de nourriture

 Jeunesse une classe d'âge

redpajama_set_name: RedPajamaWikipedia


The Kimball Family: Four Boys and a Girl: So, so long!
Sorry it has been a few days. On Monday my sister, Nicki, had a baby boy - he came about 1 month early, so last I heard the only name he had to go by was "Plant", picked by my niece, Alyson, who is 3 and who we have yet to meet!! Hopefully the name has changed, but the important part is that he seems to be doing well, by all of my parents accounts.
So, we are trying to decide the whole where to live issue. We absolutely LOVE our neighborhood - but I drive both of the boys to school, about 15 minutes each day for Jonah. More annoying than the drive, though, is that everyone at his school lives up by it, for the most part, and I'd like him to play with those kids too, to strengthen relationships. Every time he gets a note to play at someone's house, I have to explain that we live out of the neighborhood, so it is more difficult. Anyways, a friend of my BIL's came by to discuss a roofing job, and I got all excited about redoing this, putting in a backyard, etc. because the guy is a contractor so he does all of that stuff. So I'd like it if we hunkered down and put some time, $, and effort in, or started over.
A funny story from Jonah on Thursday. So I preface this with how Jory tries to instill a need to be good with the boys by telling them that they are Kimball's. "You are Kimball's - Kimball's would do (blank) or wouldn't do (blank)". We are in the car on the way home from school, and Jonah decides he wants to go to McDonald's for dinner - which is usual because we pass one on the way home. So he's saying to me "So, I want McDonald's for dinner." No response is necessary from me so he says "Yeah, Mom, we are Kimball's, and Kimball's can go where they want for dinner - Kimball's can go to McDonald's" - it was so funny!!
Jonah has been doing very good in school, and pretty much the only notes I get home say that he was very good at helping someone out or being patient while teachers dealt with a problem student. He also has been doing great at soccer. They had his last game today, and he is such a go-getter. He is just all over the place, and although he only got one goal this year - I am SOOO proud of him!! Some of his teammates have conked out mid game because the field is about double what they had last season, but Jonah just kept on trucking!! It is in the 80's today - so I have barely seen him - out on the slip-n-slide with the boys, now watching a tramp being put up in the neighbor's yard.
Eli is just running out crazy in the back with Dad and Jax. Jory had a camp out with the deacons last night and Eli and Jonah slept over at Jory's parents house. I got yardwork done - I was out there until about 12:00 midnight. It was good to get it done. So I go pick up the boys in the morning and Eli looks at me and says "I peed my pants" I looked at him and said "Right now?", he said "No, last night" - this is the first my MIL heard of it and says she had no clue. Sorry Robyn - and more laundry for Mom.
Jax slept from 7:30 last night until 8:30 this morning. It was great. He even took a good 3 1/2 hour nap - maybe he's hitting a growing spurt - not my baby! He is just hitting the back yard like no tomorrow - and will not stop going- Jory is out there with him and Eli. A good little break for me. Jory is kind of fading because everyone forgot sunscreen and they did a 5 mile hike today - he is FRIED!!
Well, better go get some grub started!! Thanks for stopping by!
I have to say that you moving is a VERY BAD IDEA!
You're thinking of moving? What!? I suppose I don't have room to talk. I love that Jonah turns that statement around on you guys because Jane does that all the time. You just can't win. Those boys are adorable and we need to have them come play on Jane's new swing set.
Do I Really Have a "Spare" Minute?
Just another day. . .


Where: Meet us at the corner of Mira Mesa Blvd. and Greenford Drive no later than 10:45am.
Info: The annual 4th of July Parade steps off at 11:30am from the corner of Mira Mesa Blvd. and Greenford Drive, traveling down Mira Mesa Blvd. to Camino Ruiz. Family Fun Day continues until 9pm, and the Park provides a front row seat for the Fireworks Extravaganza. Join the Carol Kim Campaign on the parade route!
Info: Join the San Diego County Young Dems for our July Social. Enjoy some brew while you mix and mingle with other like-minded individuals!


Can the same be said about cloud computing, today?
It is evident that cloud computing is the next major step in computing, in general. But is it the next major step in the evolution of corporate information systems? Everyone is certainly talking about it; but who is actually doing it?
According to Pew Research, anyone with a Gmail or YouTube account is participating in the cloud computing revolution. Perhaps I need to change my perspective about cloud computing in order to agree with them. The Pew report focuses on end-user adoption of cloud-based services, or SaaS. The way I see it, end-users don't engage in cloud-based computing. End-users use applications. Regardless, neither Gmail nor YouTube represent an evolution in corporate information systems.
So again, what corporations are engaged in cloud computing?
Kevin Jackson and Peter Laird have started some interesting analysis on Cloud Computing mindshare. It is not surprising that, in Kevin's analysis, the usual suspects show up in the top three (3): Amazon, Google, and Microsoft. But these three represent companies that have made huge infrastructure investments and are figuring out how to gain further returns from those investments. What they don't represent is the adoption of cloud computing by the managers of corporate information systems.
For now, at least, it looks like cloud vendors don't kiss and tell. That said, I can't wait to be enlightened.
Walter Pinson is the Chief Software Architect at SMBLive; whose IAmFound social marketing platform helps SMBs (small and medium-sized businesses) get discovered online and generate inbound sales leads. In his role as CSA, Walter manages a globally distributed product development group and has SaaS products deployed at major telecommunications channel partners in the US, UK, Switzerland, Canada, and Mexico. Additionally, Walter is a co-founder and principal of Pinson3 Systems; a boutique software engineering concern serving the intelligence and defense sectors with solutions ranging from enterprise web-based multi-modal biometric analysis systems to information warfare training systems. Prior to his role at SMBLive and founding Pinson3 Systems, Walter was a Senior Strategy Consultant with Microsoft focused on SaaS, SOA, and .NET adoption in the telecommunications sector. Walter has published articles for Cloud Computing Journal, is a member of ACM and IEEE, and currently has one patent pending. He holds a bachelor's degree in computer science from The Volgenau School of Information Technology and Engineering at George Mason University, where he focused on distributed and parallel computing. Walter is currently working on his MBA at the GMU School of Management.


text:
The Company Newsroom of
Stambaugh Ness
Social Wire
About Stambaugh Ness
2600 Eastern Blvd
York, PA	17402
Powered by Newswire
Stambaugh Ness Announces Acquisition of T. Wayne Owens & Associates, PC and TWO CPAs & Consultants, Inc. (TWO)
Combination Strengthens Firm's National Architectural & Engineering Market Leadership
Press Release - updated: Nov 9, 2020
YORK, Pa., November 9, 2020	(Newswire.com) - Stambaugh Ness (SN) announced that effective November 9, 2020, it has acquired T. Wayne Owens & Associates, PC, a certified public accounting firm headquartered in Augusta, GA, and TWO CPAs & Consultants, Inc., a consulting firm headquartered in Dublin, OH. Joining the firm as Owners are T. Wayne Owens, CPA, CGMA, CDA; Mindy L. Toole, CPA, CGMA, CDA; Brad Wilson, CMA, CDA; and Becky Carlson, CPA, ABV, CDA. In addition to the new owners, 12 employees have joined SN.
This combination strengthens SN's position as an Architecture & Engineering (AE) thought leader and the go-to national firm serving the AE industry. Delivering solutions that are anticipatory, products that are innovative, and approaches that focus on transforming businesses, SN serves over 1,000 AE clients nationally in all 50 states with team members in 19.
SN President and Chief Executive Officer Steven L. Hake states, "We are thrilled to bring together three organizations that are equally dedicated to exceptional client service and laser-focused on serving the AE industry. TWO is highly respected not only by their clients and colleagues, but as national AE authorities, particularly for their specialized expertise in government contracting.
Together, we will help our clients prepare for the NextGen through a shared desire to provide collaborative, forward-thinking, and future-ready solutions.
Steven L. Hake
Wayne Owens, Owner of both firms, says, "I am really excited about our future with SN. As we have grown, we have changed in order to live up to our tagline 'Built for A/E/C'. First by adding the expertise of Brad Wilson and Becky Carlson through TWO CPAs & Consultants, Inc. This allowed us to offer an expanded range of services focused on AE including valuations, ownership transition, and mergers and acquisitions. SN is the next step in our growth. Their offerings for Deltek, including Ajera, as well as other technical support, focused on the AE industry fulfills the vision Mindy and I shared when we started ten years ago."
In conjunction with this acquisition, SN is launching a new Government Contract Services practice area that will be led by Wayne Owens. This focus area will leverage Wayne and his team's exceptional knowledge and understanding of government contracting including the Indirect Cost Rate audits under the Federal Acquisition Regulation (FAR), as well as their deep relationships with Departments of Transportation, nationally.
Stambaugh Ness is a multi-discipline, professional services firm serving clients nationally with a strategic focus on the architecture and engineering, professional services, construction and real estate, and manufacturing industries. Through the Stambaugh Ness family of companies, clients have access to a sophisticated menu of services including compliance, business and tax advisory, technology, Microsoft solutions, Deltek ERP software, and NextGen solutions. The firm is an independent member of the BDO Alliance USA.
www.stambaughness.com
Source: Stambaugh Ness
Categories: Business News
Tags: Architecture, Business Consulting, Engineering, Federal Acquisition Regulations, Government Contracting, Stambaugh Ness, T Wayne Owens
View Website or Media Room
Stambaugh Ness is a multi-discipline, professional services firm serving clients nationally with a strategic focus on the architecture and engineering, professional services, construction and real estate, and manufacturing industries
Heather Zecher
Marketing Director, Stambaugh Ness
moc.ssenhguabmats@rehcezh
Stambaugh Ness Welcomes Kevin Hebblethwaite as Director of Business Development
Addition Strengthens SN's Position in Architecture & Engineering Space
Stambaugh Ness - Sep 7, 2021
Stambaugh Ness Welcomes Jeffrey Adams as Director of Firm's Merger and Acquisition Practice
Stambaugh Ness - Aug 18, 2021
Stambaugh Ness Announces New AE Advisory Practice Area
Stambaugh Ness - Mar 15, 2021
Get notified about new releases.
Sign up for email updates from Stambaugh Ness .
Notification Frequency
Press Releases Instant Daily Never
News Instant Daily Never
Event Instant Daily Never
Blog Daily Never
Facebook Daily Never
Twitter Daily Never


text:
Saatyaki So Seshendra Sharma : Blogs
Seshendra Sharma : A Poet's story of life
Saatyaki So Seshendra Sharma July 20, 2013 01:40 PM
The literary world is aware that my father Gunturu Seshendra Sarma, eminent poet, litterateur and scholar-critic, died on 30th may 2007. Ever since he expired, there has been no mention of his parents, family members and other personal details in the news and in the articles about him. Not only this, fictional lies are being spread and using money power one shady lady is being propagated as his wife and so on. This has been causing me, as his son, a great mental agony. That is why, through this article, I am revealing certain fundamental truths to the literary field of this country and the civilized society. I appeal to your conscience to uphold truth, justice and values of our composite culture.
Seshendra Sharma's family members are: Parents: Subrahmanyam Sharma, Ammaayamma; Wife: Janaki Daughters: Vasundhara, Revathi,Sons: Vanamali, Saatyaki. Only these two are legal heirs of Seshendra Sharma, socially and morally too.
STREET PLAY AND CIRCUS
In 1972, away from the civilized society, without the knowledge of parents and near and dear, in a far flung village called Halebeed in Karnataka a circus, a street play was staged. Let me make it clear that even after this street play my father did not divorce my mother Mrs. G.Janaki legally. He never had even a faint intention of committing such as uncivilized act. On the contrary, in all crucial Government documents he nominated my mother as his legal heir from time to time. During his long career as Municipal Commissioner with the Government of Andhra Pradesh, he retired 3 times. His first retirement came in 1975 by way of compulsory retirement for his anti estrablishment writings during Mrs. Gandhi's emergency. His second retirement came in 1983 when the then new chief minister N.R. Rama Rao's government reduced the age of service from 58 to 55 years. The third and final retirement in the year 1985 on attaining 58 years of age. On all these occasions, in all the government documents, my father Seshendra Sharma Nominated my mother Mrs. Janaki as his legal heir. This is precisely why the self contradictory 'second marriage' is a circus enacted away from the society and Laws does not recognize this type of street plays as marriage.
Seshendra Sharma's entire family complex is present in this picture. His parents, his wife JanakiSeshendra,children his two maternal uncles who are his two sisters husbands, their children on the occasion of his younger brother Rajaseskharam's marriage at Tirumala Hills(Temple Town,Tirupathi.AP:INDIA -June 21, 1962
LAKSHMI PARVATHI IN LITERATURE
N.T. Rama Rao, actor turned politicial married Ms. Lakshmi Parvathi in 1994 and subsequently in January 1995 he came to power for the second time. She used to act as an extra constitutional power and run the matters of government and the party. She developed her own coterie of cohorts and started dominating the party. After NTR was toppled by his own son-in-law, most of them parted ways with her. And the remaining touts left her for good the day NTR breathed his last. Ms. Indira Dhanrajgir has been playing the same role in Telugu literature over a period of more than 3 decades. In the guise of literature she developed her own coterie of lumpens with extra literary and money mongering elements – Tangirala Subba Rao, Velichala Kondala Rao (Editor, Jayanthi) Cheekolu Sundarayya (A.G.s Office, Hyderabad et al).
There are a couple of dissimilarities between these two instances. Afther the demise of NTR, L.P's coterie of cohorts disappeared once and for all. Whereas, in Indira Dhanrajgir's case new lumpens are entering the field with the passage of time. Squandering her late father's wealth, she is roping in new touts. Since NTR's wife Basava Tarakam passed away in 1984 and since he was old and sick NTR's marriage with LP has ethical basis and is legal completely. Whereas Indira Dhanrajgir's is neither ethical nor legal. Hence it is a street play. This is the reason why after my father's death she has been spending money on a larger scale and indulging in false publicity and propaganda. Bh. Krishna Murthy, Sadasiva Sharma (The then Editor of Andhra Prabha: Telugu Daily, presently with Hindi Milap) Chandrasekhara Rao (Telugu lecturer: Methodist Degree College) etc. are indulging in all sorts of heinous acts to propup Indira Dhanrajgir as my father's wife.
My father passed away on 30 May 2007. When our family was in grief and I was performing the 11 day ritual as per my mother's wish, the above mentioned Sadasiva Sharma went to Municipal Office on 4th June, created ruckus, played havoc telling them that he is from the Prime Minister's Office, made some 'senion officials' make phone calls to the officials concerned and got my father's death certificate forcibly issued. When the entire family as mourning the death of the family head, a stranger and a lumpen S.S. – Why did he collect my father's death certificate forcibly from the municipal authorities? Whom did he collect it for?
THREE NAMES OF THE SAME PERSON IN 3 DECADES
This is perhaps for the first time that the name of a lady appears in 3 forms at a time. Perhaps in 1970, in my father's collection of poems "PAKSHULU" her name appeared As Rajkumari India Devi Dhanrajgir. In 2006 she published a fake version of Kamaostav (Rewritten by a muffian Called Chandrasekhar Rao). In this book her name appears as Rajkumari India Devi Dhanrajgir Prior to 1970 in Maqdoom Mohiuddeen's (Renowned Urdu Poet) anthology of poetry 'bisath-E-Raks', in Urdu as well as Hindi, at the end of two poems her name appears as Kumari Indira Dhanrajgir. On 15th June 2007 A.P. State cultural affairs department and Telugu University jointly held my father's memorial meeting. Indira Dhanrajgir hijacked this meeting by issuing her own commercial advertisements in English and Telugu dailies. In these advertisements her name appeared as Smt. Indira Devi Seshendra Sharma and again in the commercial public notices made by her in the month of November 2007 her name appeared as Rajkumari devi etc. Why does her name appear in different forms on different occasions? Will Indira Dhanrajgir explain? Will Sadasiva Sharma clarify, who forcibly took my father's death certificate after four days of his death? Or will Bh. Krishna Murthy clarify?
If Indira Dhanrajgir has even an iota of regard, respect for or faith in love, or relation, the institution of marriage, immediately after 'Halebeed Circus', she would have used my father's family sir name and her name would have appeared as Gunturu Indira. Since she was conscious of her goal during all times and conditions she did not take such a hasty and mindless step of change of her name.
WHERE DOES THE REAL SECRET LIE?
Her life is totally illegal, anti-social and immoral.
Indira Dhanrajgir's father performed her marriage with SrikishenSeth, Nephew of the then Prime Minister to Nizam, Maharaja Kishen pershad in 1945. On the day of marriage itself Indira Dhanrajgir beat SrikeshenSeth up and ran away from him. She did not stop at that. She propagated among his friends and relatives and near and dear that he was not enough of a man and unfit for conjugal/marital life. She filed a divorce case against him and dragged it till 1969/70. Lion's share of her husband's life got evaporated and was sapped completely by then. His parents used to approach Indira Dhanrajgir's father and plead with him to preavail upon his daughter, put sense into her head and see that she either lives with their son or dissolves the marriage legally so that they can remarry off their son. But Indira Dhanrajgir did not heed. Raja Dhanrajgir after getting disgusted with her nasty activities stipulated a mandatory condition in his will. He stated that Indira Dhanrajgir would be entitled to get a share of his property only if she is married.
This is the reason why INDIRA DHANRAJGIR who has no respect for the institution of marriage or regard or desire for marital life, in the guise of love and love poetry inflicted indelible blemish on the institution of marriage which is unprecedented in the literary history of the world. After my father's death she has been indulging in more rigorous false publicity along with her coterie of touts.
KAMOSTAV: STORY OF INDIRA DHANRAJGIR'S SOUL:
With this novel Kamostav, father's literary life came to an end for good. He did not produce literary works worth mentioning in his later phase of life. During those days he asked for my opinion on that novel. I told him clearly that it lacks the form and content of a novel; it does not have a story line, plot, sequences, characters and eventually a message which every novel gives. Hence it is a trash. Several people went to court and got its publication in a weekly stopped. INDIRA DHANRAJGIR got this very trash rewritten completely by Chandrasekhara Rao and printed it. This kind of heinous development has never taken place in the recorded history of Telugu literature till date. A writing which brought disrepute to my father in the literary field and isolated him in the society, why did she get it rewritten by somebody and publish it claiming copyright to be hers? What is her motive? What is her aim? That is why Kamotsav is INDIRA DHANRAJGIR's biography, story of her inner soul.
SESHENDRA'S COPYRIGHTS:
My father gifted away copyrights of his entire works along with their translations to me by way of birth day gift to me on 2.12.1989. Since then I have published several of his works during his lifetime itself. Kamostav, the version that is secretly made available is the dirty work of cheapsters and lumpens under the leadership of INDIRA DHANRAJGIR. It is much worse than violation of copyrights. That is the reason why I have been reluctant to take action so far. If she and her debased henchmen try to violate copyrights of my father's works bequeathed to me, I shall take exemplary legal action against them.
INDIRA DHANRAJGIR made 2 public notices to the effect that my father cancelled all his earlier transfer of copyrights and retransferred all his rights to her. This is a palace intrigue in the modern era in our civilized society.
WHAT DOES LAW SAY ABOUT COPYRIGHTS?
An author can transfer copyrights of his works to any one as per her/his wish. But the Copyrights Act 1957 and the Supreme Court in its various judgments has clearly stipulated a procedure to revoke earlier assignment and transferring of copyrights to somebody else subsequently. The author has to issue a notice to the 1st assignee, giving 6 months time for reply. Depending on the reply the author can take his next step. Where as in my father's copyrights matter he did not even inform me orally of any such cancellation. INDIRA DHANRAJGIR claims that she has a typed document of transfer of copyrights signed by my father on 5.1.2006. Between 5.1.2006 and 30.5.2007, leave alone issuing a notice, he did not even inform me orally.
Seshendra with Wife Janaki this photo was submitted in the Government Documents in which he nominated her as his legal heir. 1975
My father who assigned copyrights to me in his own handwriting, when he was relatively young and physically fit did not require to cancel the 1st assignment when he was totally dilapidated, almost bedridden and was counting his days. Another important aspect of the matter is that I have printed the Xerox of my father's document in his own works as early as 1995 and have been doing so from time to time during his life time. Where as INDIRA DHANRAJGIR claims to possess a document after my father's death and she has not made it public so far. Indira Dhanrajgir tried to get my father's complete works published in different languages by Telugu University (Hyderabad: A.P.: India) by paying them Rs. 6 lakhs. I approached Telugu University and apprised them of facts. On the advice of legal experts, they stopped this project and returned Indira Dhanrajgir's money to her. It is an incontrovertible fact that Indira Dhanrajgir's document is a forged and fraudulent document which does not stand scrutiny before law. Court shall certainly award her exemplary punishment. In all societies and times literature has been social wealth/public property from time immemorial. It should not be used as a mask to grab share of parental property illegally and unethically. I am committed to this cause/ideal and appeal to the civilized society to strengthen my hands in this endeavor. INDIRA DHANRAJGIR's younger brother Sri Mahendra Pratapgir is the lone leage heir apparent of that family and keeping him in dark, she is squandering her father's wealth in Telugu literature for her nasty propaganda.
FATHER PASSED AWAY:
In 1997 when he suffered the 1st heart attack he was half-dead. Dr. Sudhakar Reddy, cardiologist of Medicity Hospitals (Native of Warangal, A.P.) performed angiogram and diagnosed that he had blocks in arteries and one valve was damaged completely. He advised open heart surgery. But INDIRA DHANRAJGIR averted it and got angioplasty performed. His health declined rapidly since then and was leading the life of virtually an invalid till he breathed his last. He suffered inexplicable mental and physical torture for about a decade. During the last leg of his journey he was isolated from his family completely. He was deserted by one and all in the literary field. When his younger brother passed away, his younger sister passed away he did not visit his ancestral home in his village and call on those families. He became target of jealousy and animosity in the society. He became a victim of false impression with the society that he was an aristocrat and rolling in luxuries. Whereas, he was deprived of even his native vegetarian food for decades together. As a silent and helpless witness to these painful happenings, I was subject to untold mental agony.
Seshendra's last metrical poem on his son Saatyaki
In the later half of March 2007 on one of my visits to him, I was aghast at his condition. His entire body was swollen. His appearance was like that of a stuffed gunny bag. I told him to get hospitalized. I told INDIRA DHANRAJGIR to rush him to a hospital. But of no avail. On 30th May 2007 at about 11 pm I got a phone call from her "Come soon/Serious" she said. As I entered at 11.15 pm "Go inside/he is no more" she said.
"One day when swarms of lamps vanish, in the light of a lonely lamp I ask the dumb pillars".
"Can't you liberate me from the disgust of this existence?
I ask those stand still forest flame trees
Which blossom flowers at that very place year after year" can't you rescue me?
I ask those high roof tops and this Venetian furniture
Which every one feels are greater than me,
"can't you rescue me from the disgust of this existence?"
All these answer in a melancholic voice
"We have been languishing since more than 100 years watching the same unchanging scenes we are older prisoners than you are" (Janavamsham: Telugu: Seshendra: Page 80-81:1993: Translated by me)
My father's first biography (in Hindi) titled "Rashtrendu Seshendra: Ashesh Aayaam" by Dr. Vishranth Vasishth appeared in 1994. Touching upon these very sensitive aspects of my father's life he commented in that book "SONE KE PINJRE ME PANCHCHI" (A bird in a golden cage). Alarmed and agonized by his rapidly declining health as early as June 2002, in order to bring pressure on INDIRA DHANRAJGIR, I gave a 2 cassettes long interview to Vijayaviharam of Janaharsha group. Later on when I enquired about that interview they said that in the raids conducted on their premises, they got destroyed.
I wanted to rescue my father and bring him back home when he was in good health. Alas! At last, I took him to the burial ground, laid him on the funeral pyre and consigned him to flames and returned home all alone.
G. Satyaki
S/o Late G.Seshendra Sharma
E-mail: saatyaki@gmail.com
"Kaasyashcha parameswasaha sikhandeecha mahaarathaha
Drishtadyumno viraatascha Saatyakischaaparaajithaha"
-Bhagavat Gita-Chapter-1/17 sloka
PARDON ME FATHER
I could not rescue him from the clutches of that nymphomaniac and vampire. There may be an exception or two but an average Indian woman desires from the depths of her soul that her husband should live long and she should pass away before him. She performs prayers and fasts on auspicious days for this purpose. She in spite of being 3years elder to him did away with my father in a planned and premeditated manner and I was a silent and helpless witness to it. He suffered 1st Heart attack in November 1997. Cardiologists performed angiogram and advised open heart surgery. Because there were blocks in vessels and one valve was damaged. But she successfully thwarted it and without my knowledge or informing any one got angioplasty done in Mediciti (Hyderabad: AP; India) her plan was to do away with him and live long, and establish herself as his wife through his books. He was succumbing to her blackmail. My overwhelming hunch is that she was threatening him with social insult and humiliation if he parts ways with her. Between 1997-2007, she played football with his body. He used to be hospitalized every now and then with swollen body and heart pain. Because of damaged valve pumping was impaired and water used to accumulate in the system. Every time I used to force her to hospitalize him. He used be in ICCU for a couple of days and recover marginally. After each visit to hospital he was getting debilitated gradually. He was put on wheel chair. He was virtually under house arrest. He was not allowed to speak to friends and family members. Visitors were kept away. He was taking Lasix (Tablet: is a diuretic that is used to treat fluid accumulation, caused by heart failure, cirrhosis, chronic kidney failure, and nephrotic syndrome.) to flush out water accumulated in his body. This creates a painful dilemma in me whether my interference in his health matters was just. As his son it was my moral duty to protect him. But I sometimes feel if I were not to interfere she would have put him to death long ago and thus he would have escaped from physical and mental torture quite early.
Towards perhaps end of the month of March she withdrew medication. He got swollen suddenly and that condition continued till the last day i.e. 30th may 2007. Each time I visited I used to tell that witch to take him to hospital. But after a couple of visits I got convinced that she made up her mind this time to do away with him. I requested a bastard who was feigning to be a friend of mine, who incidentally happens to be a legal luminary of this region to send a doctor friend to that place and ascertain the exact condition of his health. But of no avail.
I kept on telling him to come out of that place and lead a normal and healthy life. Her blackmail gained an upper hand and I lost in my efforts to restore health to him and bring him back to civilized society. O God pardon me for not being able to outmaneuver her machinations. Pardon me father.
Saatyaki So Seshendra Sharma
Media/Journalist/Author
Andhra Pradesh , INDIA
Recent Blogs By Author


The interior of the Explorer delivers as much as 151.5 cubic feet of passenger volume and up to 81.7 cubic feet of cargo capacity.
Explorer offers access to the latest connectivity features like voice command, navigation, and smartphone integration with SYNC 3 technology.
Available features like a power hands-free liftgate, USB smart-charging outlets, keyless entry, and more make getting on the rod a snap.
The 2018 Ford Explorer interior is designed to add a new level of comfort and convenience to your daily drive, whether that means a leisurely trip to downtown Mascoutah, or venturing into the wilds of St. Louis, you'll have everything you need to get there in comfort and style.
It's easy to get everything you want for your next adventure when you step up to a new Ford Explorer.
Schedule a test drive of the 2018 Ford Explorer at Auffenberg Ford Belleville and experience intelligence in action.


It is about setting and then managing expectations.
Our resources and approach are specifically aligned to your project's success.
We will assign seasoned resources. We will prepare the business for the work ahead. We will facilitate communications.
We will drive project progress and support course correction. Moreover, we will communicate in your language as a business user, executive sponsor, or application developer. With a common understanding, we will succeed.


text:
Vanderbilt adds Ohio State, Michigan, junior-college transfers
Vanderbilt football announced three transfers added to the roster on Wednesday.
Michigan graduate transfer long-snapper Scott Sypniewski and junior-college transfer offensive lineman Saige Young will be eligible immediately this season. Wide receiver Alex Stump, an Ohio State transfer, must sit out the 2017 season to fulfill NCAA transfer rules before playing in 2018.
Sypniewski played in 36 games over three seasons as Michigan's long-snapper. He graduated from Michigan in the spring, and he will have one year of eligibility remaining. He will help fill the void by the graduation of Vanderbilt's two long-snappers from last season.
Young, a 6-foot-4, 315-pounder was an all-conference guard at Dodge City (Kan.) Community College. He has three years of eligibility remaining.
Stump took a redshirt at Ohio State in the 2015 season. He made one reception in 2016.
"Our team immediately becomes a better and deeper football squad with the addition of these young men," Vanderbilt coach Derek Mason said. "Their arrival on campus gives our program a full roster entering preseason camp."


text:
Home Movies Journey's End
The Lord of the Rings: The Return of the King (2003)
Nausicaä of the Valley of the Wind (1984)
20,000 Leagues Under the Sea (1954)
Rio Bravo (1959)
Seven Samurai (1954)
Chocolat (2000)
Journey's End (2017)
Set in a dugout in Aisne in 1918, a group of British officers, led by the mentally disintegrating young officer Stanhope, variously await their fate.
Release : Dec 14, 2017
Runtime : 108 minutes
Language : Español
Studio : Fluidity Films, BFI, Wales Screen, British Film Company, Metro International Entertainment, Third Wednesday Films, Anyway Productions, uMedia
Cast : Asa Butterfield, Sam Claflin, Paul Bettany, Tom Sturridge, Toby Jones, Stephen Graham
Crew : John Hubbard (Casting), Ros Hubbard (Casting), Anushia Nieradzik (Costume Design), R.C. Sherriff (Novel), R.C. Sherriff (Theatre Play), Saul Dibb (Director)
Genre : War, Drama
Keyword : france, world war i, british army, army, alcoholism, alcoholic, trenches, 1910s
Search Related » Journey's End «
Journey's End 2017 Full HD quality
Watch Online Journey's End 2017 Full Movie
Journey's End Full Movie Free
Download Journey's End Full Movie HD
Journey's End Full Movie Watch Online Free
Journey's End Full Movie Download
Journey's End 2017 Full Movie
Journey's End 2017 Streaming
HD Version For Journey's End 2017
Journey's End 2017 Full Length Movie
Journey's End 2017 Movie trailer
Journey's End 2017 Movie Wiki


= [[Funding Transaction]] = The Funding Transaction creates the channel. During this stage, funds are sent into a multisig address controlled by both Alice and Bob, the counterparties to the channel. This address can be funded as a single-payer channel or by both Alice and Bob. The [[Funding Transaction]] creates and signs a total of 3 transactions but only broadcasts one. The one transaction that is broadcast funds the channel, and spends to a multisig output. It is signed by whomever chooses to fund the channel, Alice and/or Bob. When this [[Funding Transaction]] receives sufficient confirmation, the channel is considered open. The number of confirmations needed to be considered "sufficient" is configurable in the [[Lightning Network]]. The remaining two transactions that are created but not published are [[Commitment Transaction|Commitment Transactions]]. These are signed by both payment channel peers, e.g. Alice and Bob, and each peer stores its own [[Commitment Transaction|Commitment Transactions]] for later use in a possible future [[Closing Transaction|Closing Transaction]].


ABILITY English is all about unforgettable people, and the teachers you will meet here are very skilled at what they do.
The teachers at ABILITY English have received awards and been invited to present at important English language teaching events and conferences. This is because of their dedication and commitment to being the best English language teachers they can be.
When you arrive at ABILITY English it will be obvious to you that our teachers genuinely care about helping you achieve your study goals and being part of your Australian experience.