text:
\section{Discussion}
Without fear of aligning an overused {\em clich\'e}, the release of
large language neural models have not only revolutionized the NLP
field by bringing a major leap in performance in almost every tasks
they were applied to, they crucially changed the perception of the
risks of their potential misuse. The point is that this dramatic boost
of performance has led the field to rely on the capacity of those
large models to transfer their, in layman's terms, ``knowledge'' to
other tasks via various transfer learning modalities. Yet, with this
transfer, the potential data biases inherent to large corpus
collection used for pre-training are also susceptible to appear.
\newcite{gehman-etal-2020-realtoxicityprompts} thoroughly demonstrated
that all generative language models they tested (from GPT1 \cite{gpt}
trained on Book Corpus only to GPT3 \cite{gpt3} and CTRL
\cite{Keskar2019CTRLAC} trained on various corpora, including
user-generated content and web-crawled data sets) were capable of
producing toxic output in specific conditions\iftrue and presented
different ways of alleviating this behaviour\fi. Having been
pre-trained on Common Crawl-based corpora, our models are certainly
not immune from toxic content generation.

\iffalse {\em This is confirmed by a small
scale experiment we carried out to assess the toxicity of 360
generated sequences from 6 seed words susceptible to ignite toxic
content when used in a prefix prompt (eg. Les arabes sont .. {\em
  (arabs are...)}, les juifs sont... {\em (jewish are..)}, etc.). We
slightly varied the formulation (4 different at most) for each word.
The toxicity assessment was done automatically via the {\sc Google
  Perspectives
  API}\footnote{\url{https://github.com/conversationai/perspectiveapi}}
and we considered as toxic anything the {\sc Perspectives API}
predicted as toxic with a probability higher than 0.7.\footnote{We
  first used a 0.5 toxicity probability threshold but it led to many
  false positive where the simple presence of a word such as 'noir' or
  'arabe' (arab and black people) seemed to trigger the classifier.
  For example, sentences such as ``Les Noirs et les Maghrébins sont aussi victimes
  des violences policières.'' {\em (Black people and arabs are also victims
  of police violences)} or ``Les Noirs sont considérés comme des êtres
  inférieurs et que la femme noire doit être l'égale de l'homme noir :
  cette différence de traitement est d'un racisme évident...'' {\em (Black
  people are considered as inferior beings and that  black woman
  must be equal to black man: this difference is blatant racism...)} were labeled as toxic by the {\sc
Perspectives API } classifier.}

Our results in Table~\ref{tab:toxic} show that even with a curated
corpus such as {\sc CCnet}, the model still exhibits some output
labeled as toxic in racial and religious categories (4.15\%, 15 out of
360 generated
sequences). \fi \newcite{gehman-etal-2020-realtoxicityprompts} explored
two main ways to filter toxic outputs, one being based on the pursuing the pretraining on
less toxic data sets or on toxicity-annotated data set; the other
focusing on  the alteration of the decoding
strategy. \newcite{Krause2020GeDiGD} proposed to guide the output with
smaller language models. Given the interest in generative models, this
is an active line of research that we are currently pursuing.

More generally, the question of knowing whether the
pre-training data should be curated more or should the model output,
depending on the downstream application in sight, be debiased, or filtered, directly  is still
the object of vivid debates among the community
\cite{Bender_et_al:2021:Parrots,goldberg:2021:reparrots}, while of
course there is an agreement toward responsible use of such
technology.

In this aspect, the release of a GPT-generated text detector by
\newcite{Antoun2021AraGPT2PT} along their Arabic GPT2 model is an interesting
step toward this direction.

Regarding the environmental aspects of this model, our model pretraining
experiments consumed about 62k gpu hours from the Jean Zay HPC cluster. Being
based in France, its energy mix is made of nuclear (65-75\%), 20\%
renewable and the remaining with gas (or more rarely coal when
imported from abroad) (S.Requena, Dir. of Jean Zay, P.C).

Regarding the performance of our models which almost constantly
outperform their closest French counterparts, the
GPT\textsubscript{fr} models \cite{simoulin2021modele}, one explaining
factor could be the size of our pretraining data set (30B token vs
3B). Given our computing limitation, we choose from the beginning to
use an experimental protocol as comparable as possible to the one used
for the CamemBERT model evaluation \cite{camembert}, it would of
course be interesting to perform a head to head comparison with the
GPT\textsubscript{fr} model. In terms of raw performance, it has been
constantly reported that GPT-based models on the billion parameter
scale provide inferior performance when compared to their ``regular''
transformer counterparts in classic fine-tuning scenarios, our results
confirm this for French as well while highlighting the interest of our
models in generation-oriented scenarios (such as text summarization
where PagnolXL establishes a new state of the art for French). As for
English \cite{lester2021power}, our encouraging preliminary
prompt-tuning results suggest that this approach is promising and
could be a way to close this performance gap.

\iffalse

\begin{table}[h!]
\begin{center}
{\footnotesize
\begin{tabular}{rcc}
\hline
{ Les X sont (X are)} & { \% of toxic} & { toxic \% overall} \\
\hline
arabes (arabs) & 13.56 & 2.22 \\
juifs (jewish) & 5 & 0.83 \\
musulmans (muslims) & 3.39 & 0.55 \\
noirs (black people) & 3.33 & 0.55 \\
femmes (women) & 0 & 0 \\
handicap\'es (disabled) & 0 & 0 \\
\hline
total &  & 4.15 \\
\hline
\end{tabular}
\caption{Detected toxicity of PAGNol-XL (top-k,k=10) generated sentences with prompt of the form ``x are''
  }
\label{tab:toxic}

\end{center}
\end{table}

\fi

\section{Introduction}

Large pre-trained language models are the workhorses of modern Natural Language Processing (NLP). The use of scalable and efficient attention-based Transformers \cite{vaswani2017attention}, rather than recurrent neural networks, has enabled increasingly large and capable models. Through self-supervised learning, these models learn contextual word embeddings, building a general representation of language. After this \emph{pre-training} they can be \emph{fine-tuned} to target specific tasks (e.g. classification, parsing, summarization).

Three approaches dominate the field: (1) causal autoregressive decoder-only models, such as GPT \cite{gpt}, learning from a general language modelling tasks; (2) bidirectional encoder-only models, such as BERT \cite{bert}, learning from masked language modelling; (3) sequence-to-sequence models, such as BART \cite{bart} or T5 \cite{raffel2020t5}, combining both a bidirectional encoder and an autoregressive decoder, learning from a language denoising task. Encoder-only and sequence-to-sequence models excel in language understanding tasks, and have shadowed autoregressive models as a lesser option.

Autoregressive models have been shown to predictably benefit from increased size \cite{kaplan2020scaling,henighan2020scaling}. Scaling laws establish a direct relationship between model size and end-task performance, justifying the training of increasingly large models \cite{gpt3,zeng2021pangu,kim2021changes,wei2021finetuned}. These laws can also inform design decisions, helping practitioners use their available compute budget optimally. A significant finding has been that larger models are more sample and compute efficient: with a given compute budget, it is preferable to train a larger model significantly short of convergence than to train a smaller model to convergence. Furthermore, at extreme-scale, such as the 175 billion parameters of GPT-3 \cite{gpt3}, autoregressive models exhibit unique few-shot abilities: they can learn from a few prompted examples, without weight updates. This capability questions the current fine-tuning paradigm, and may make billion parameters models more attractive and usable. Recent forays into \emph{prompt engineering/tuning} \cite{li2021prefix,lester2021power} have even seemingly bridged the gap between few-shot performance and fine-tuning.

Encoder-only (CamemBERT \cite{camembert} and FlauBERT \cite{flaubert}) and sequence-to-sequence models (BARThez \cite{barthez}) exist for the French language, and recently a decoder-only model with 1 billions parameters has been made available \cite{simoulin2021modele}. We introduce PAGnol\footnote{PAG: \emph{Pré-Apprentissage Génératif}. Marcel Pagnol was a famous French novelist.} in this family, a collection of four French GPT-like models, and make the following contributions:
\begin{itemize}
    \item \textbf{Largest French model.} We train on CCNet and publicly release four models, with up to 1.5B parameters for PAGnol-XL. At the time of this work, this is the largest non-sparse French language model available, and we plan to explore increasingly large and powerful models in the future.
    \item \textbf{Optimal scaling.} We use scaling laws to inform our training setup, resulting in optimal use of our compute budget. PAGnol-XL is trained with a budget of only 3 PF-days, just as much as the 13 times smaller CamemBERT. From our collection of models, we adjust scaling laws for the French language.
    \item \textbf{Dataset suitability.} We highlight the importance of proper dataset pre-processing when training generative autoregressive models. While OSCAR has been relied on for French encoder-only models, we find it is not suited to PAGnol, leading to low-quality offensive outputs.
    \item \textbf{End-task performance.} We evaluate on discriminative (FLUE) and generative tasks (question answering on FQuAD and summarization with OrangeSum) in the fine-tuning and prompt tuning regimes. We establish a new state of the art for summarization in French on OrangeSum.
\end{itemize}

\section{Related work}
\paragraph{Language models.} The design and training of neural language models able to create and process word embeddings is the cornerstone of modern NLP. Early on, self-supervised learning was identified as an efficient and scalable way to train such models. The use of deeper and more complex neural architectures enabled going from static embeddings (\textit{word2vec} \cite{mikolov2013efficient}, \textit{GloVe} \cite{pennington2014glove}) to contextual embeddings, allowing models to deal with polysemy. Although approaches such as ELMo \cite{peters2018deep} and ULMFiT \cite{howard2018universal} highlighted that learned representations can be transferred across downstream tasks, the poor scalability of RNNs prevented this vision from being fully realized.

By getting rid of the costly and delicate recurrent processing, attention-based Transformers \cite{vaswani2017attention} spurred a wide interest in NLP. GPT \cite{gpt}, a decoder-only variant of Transformers, demonstrated large-scale transfer learning from general language modelling to 12 NLU tasks. Along with the rise of easy-to-use libraries, encoder-only BERT \cite{bert}, relying on masked language modeling, made NLP a commodity -- wherein every practitioner could rely on a pre-trained language model and fine-tune it cheaply to a task of interest. BERT models are limited by their ability to only "fill-in-the-gap" for a span of words: this forbids their use in generative tasks (e.g. summarization).

With sequence to sequence models and pre-training through denoising tasks, the original architecture of Transformers made a comeback with BART \cite{bart}, bridging the gap between the generative capabilities of decoder-only models and the downstream task performance of encoder-only models. Through gradually larger and more powerful architectures, state-of-the-art models are approaching human-level performance on many tasks.

Successive generations of GPT models have questioned the current fine-tuning paradigm. GPT-2 \cite{gpt2}, with 1.5 billion parameters, demonstrated that large language models could tackle entirely new tasks through \emph{few-shot learning}\footnote{In other areas of machine learning, this has been referred to as \emph{zero-shot learning}, as no weight updates are necessary.}. Without any fine-tuning, from just a few prompted examples, GPT-2 achieved fair performance on a number of complex downstream tasks. Furthering this endeavour, GPT-3 \cite{gpt3}, with 175 billion parameters, achieved state-of-the-art performance on some tasks, without the need for fine-tuning. This opens new possibilities for low-resources tasks, as well as paths to more natural interactions with these models: recent research suggests the gap between few-shot learning and fine-tuning may even be bridged through so-called prompt programming/tuning \cite{li2021prefix,lester2021power}.

\paragraph{Scaling laws.} More specifically to our setting, neural language models have been shown to predictably benefit from increased scale \cite{kaplan2020scaling}. Their training dynamics are size-invariant, allowing test loss, parameter count, and dataset size to be correlated through smooth scaling laws. This is in fact true of all GPT-like autoregressive models, even when applied to image, multimodal, or mathematics modeling \cite{henighan2020scaling}. Gains in autoregressive cross-entropy loss also directly translates to gains in end-task performance after fine-tuning. As they relate to compute budget, these predictions can be used to inform the training of large models.

\paragraph{Non-English generative models.} BERT-like models are now available in a broad number of languages, either as specialized models are as multilingual ones. This is less so the case for generative models, perhaps because of issues in controlling the language used at generation time. For the French language, GPT\textsubscript{fr} is an autoregressive generative model, and BARThez \cite{barthez} targets some generative abilities. Smaller-scale efforts exist, such as BelGPT \cite{louis2020belgpt2}, but they are limited to small models. GPT models have been trained for German \cite{stefan_schweter_2020_4275046}, Chinese \cite{zeng2021pangu}, and the Arabic language \cite{antoun-etal-2021-aragpt2}, among others.

\section{Efficient training with scaling laws}

\paragraph{Scaling.} We use scaling laws to inform the duration of the training of our largest models. Rather than training to convergence, which would be wasteful, we train to optimality, as predicted by the equations provided in \cite{kaplan2020scaling}. This is akin to what has been done for GPT-3, and this enables us to keep our computational budget in line with that of CamemBERT, a model 13x smaller than PAGnol-XL. We find that training all of our models for a single epoch on the 30GT of CCNet enables us to reach optimality for the most expensive XL model. Table \ref{tab:architectures} presents the ratios between the compute budget effectively used and that to optimality ($r^C_\text{opt}$) or to convergence ($r^C_\text{conv}$). While our small model is trained to convergence,  others are trained significantly short of it. We find that our training performance matches nicely with the estimated $2.6$ PF-days for the training of GPT\textsubscript{fr}-LARGE from \cite{simoulin2021modele}.

\section{PAGnol}
In this section, we describe the data, model, and training specifications for PAGnol. In Table \ref{tab:comparison}, we highlight some of the key differences and similarities with other French models, and in Table \ref{tab:multilingual} we present two multilingual models that we consider in the following.

\begin{table*}[t]
\begin{minipage}{\linewidth}
\centering

\scalebox{0.7}{\begin{tabular}{@{}lcccc|c@{}}
\toprule
                                  & \textbf{CamemBERT} & \textbf{FlauBERT} & \textbf{BARThez}  & \textbf{GPT\textsubscript{fr}}                                                             & \textbf{PAGnol (ours)} \\ \midrule
\textbf{Language}                 & French             & French          & French            & French                                                               & French                 \\
\textbf{Parameters}               & 110/335M           & 138/373M            & 165M              & 124M/1B                                                                       & 124M/355M/773M/1.5B    \\
\textbf{Context}               & 512           & 512          & 768              & 1024                                                                       & 1024/2048    \\
\multirow{2}{*}{\textbf{Dataset}} & OSCAR              & Custom\footnote{\label{fnote:data}FlauBERT and BARThez use a similar pre-training dataset assembling CommonCrawl, NewsCrawl, Wikipedia, and other smaller corpora.}       & Custom$^\text{\ref{fnote:data}}$           & Filtered Common Crawl                                                                       & CCNet                  \\
                                  & 33GT/138GB         & 13GT/71GB                & 66GB              & 1.6/3.11 GT & 32GT                   \\
\textbf{Tokenization}             & SentencePiece 32k  & BPE 50k & SentencePiece 50k & BPE 50k                                                         & BPE 50k                \\
\textbf{Compute [PF-days]}        & 3/10               & $\sim7/26$\footnote{\label{fnote:compute}Insufficient data was provided by authors to infer compute budgets properly.}            & $\sim4^\text{\ref{fnote:compute}}$    & ?/2.6                                                           & 0.3/0.7/2/3            \\ \bottomrule
\end{tabular}}

\caption{Model, data, and training setup for PAGnol and other French models. Data size is reported in gigatokens (GT), and compute in PF-days ($8.64 \times 10^{19}$ FLOP). PAGnol is the largest French model. Despite being significantly larger than existing models, its compute cost remains reasonable: as recommended by scaling laws, we train models to optimality, and not to convergence.}
\label{tab:comparison}
\end{minipage}
\end{table*}

\begin{table}[t]
\centering
\scalebox{0.7}{\begin{tabular}{@{}lcc@{}}
\toprule
                  & \textbf{mBERT} & \textbf{mBART} \\ \midrule
\textbf{Language} & 104 languages & 25 languages \\
\textbf{Parameters} & 110M        & 610M \\
\textbf{Context} & 512           & 768   \\
\multirow{2}{*}{\textbf{Dataset}} & Wikipedia & CC25\\
                                 &            & \begin{tabular}[x]{@{}c@{}}180GT/1369GB\\(10GT/57GB French)\end{tabular}  \\
\textbf{Tokenization}             & WordPiece 110k  & SentencePiece 250k \\
\textbf{Compute [PF-days]} & 2  & $\sim60^\text{\ref{fnote:compute}}$ \\ \bottomrule
\end{tabular}}
\caption{Model, data, and training setup for multilingual models including French that we consider. Data size is reported in gigatokens (GT), and compute in PF-days ($8.64 \times 10^{19}$ FLOP).}
\label{tab:multilingual}
\end{table}

\subsection{Pre-training data}

\paragraph{Sourcing.} The Common Crawl (CC) project browses and indexes all content available online. It generates 200-300 TiB of data per month (around 5\% of which is in French), and constitutes the bulk of most NLP datasets nowadays. We consider in our experiments two datasets based on CommonCrawl data: CCNet \cite{wenzek2020ccnet} and OSCAR \cite{oscar}. Once tokenized, OSCAR contains 33GT and CCNet 32GT. We use CCNet for all our main experiments and released models, and compare with results obtained on OSCAR in Section \ref{sec:oscar}. We validate on the fr-wiki dataset (0.5GT) and French TreeBank (650kT) \cite{abeille2003building}.

\paragraph{CCNet.} CCNet combines the usual fastText \cite{joulin2017bag} pre-processing of CC data with an additional filtering step to select high-quality documents. This filtering is done through a language model trained on Wikipedia, ensuring a text quality similar to that of its articles. We use a version of CCNet identical to the one considered in the CamemBERT paper.

\paragraph{OSCAR.} OSCAR uses a fastText classifier to select documents and identify their languages, without any additional filtering. OSCAR is thus more varied, but more "noisy", than CCNet. OSCAR has been used to train other French language models such as CamemBERT.

\paragraph{Tokenization.} We use byte-level Byte-Pair Encoding (BPE), with a vocabulary of 50,262 tokens: 256 bytes, 6 special tokens, and 50,000 merges. Paragraphs are separated by an <EOS> token and documents are separated by a <SEP> token. We add a prefix space before tokenization, so that the first word in a paragraph is tokenized in the same way as if it was at any other position. This is similar to the setup of FlauBERT and GPT-2. For the models trained on OSCAR, we use a slightly smaller vocabulary size of 50,000 tokens.

\subsection{Model specification}
\begin{table}[h]
\centering

\scalebox{0.7}{\begin{tabular}{@{}cccccccc@{}}
\toprule
\textbf{PAGnol}      & $n_\text{params}$ & $n_\text{layers}$ & $d_\text{model}$ & $n_\text{heads}$ & $C$ [PF-days] & $r^C_\text{conv}$ & $r^C_\text{opt}$ \\ \midrule
\textbf{S}  & 124M              & 12                & 768              & 12               & 0.3           & 1,3              & 9,0               \\
\textbf{M}  & 355M              & 24                & 1024             & 16               & 0.7           & 0,5              & 3,0               \\
\textbf{L}  & 773M              & 36                & 1280             & 20               & 2             & 0,4              & 2,5               \\
\textbf{XL} & 1.5B              & 48                & 1600             & 25               & 3             & 0,2              & 1,3               \\ \bottomrule
\end{tabular}}
\caption{Model and training budgets for PAGnol. All models are trained on a single epoch of our 32GT CCNet-curated data. $C$ is the compute budget used for the training of the model. $r^C_\text{opt}$ is the ratio between $C$ and $C_\text{opt}$, the optimal compute budget derived from scaling laws. $r^C_\text{conv}$ is the ratio between $C$ and $C_\text{conv}$, the compute budget derived from scaling laws to train the model to convergence.}
\label{tab:architectures}
\end{table}

PAGnol is a decoder-only autoregressive transformer, \draftHL{reproducing the architectural choices of GPT-3, with up to 1.5 billions parameters.}{Shouldn't this be GPT3? which parts of the models are from gpt2 and which parts exactly are from gpt3? -ds  Does this look ok now? I also added a part on rotary embeddings at the end of this paragraph. The equations are a bit heavy, not sure we want to include them -ip}. We evaluate four model sizes: small, medium, large, and extra-large, with architectures detailed in Table \ref{tab:architectures}. We use a context size of 1,024 tokens for the S, M and L models. The XL uses a context size of 2,048, the largest at release for a French model. Additionally, we use Rotary Embeddings \cite{su2021roformer} in place of Learned Positional Embeddings for the XL model, since they provide much better training stability at the billion parameters regime.

\subsection{Pre-training}

\paragraph{Training objective.} We use an autoregressive language modelling objective, where the model learns to predict the next word in a sentence. To improve efficiency, we always fill the context with as much text at possible, and inform the model about separate documents through the <SEP> token.

\paragraph{Optimization.} We use the Adam optimizer \cite{kingma2014adam} with a warmup followed by cosine decay learning rate schedule. We find that proper initialization is key to training stability, and reproduce the setup effectively implemented by Megatron-LM \cite{shoeybi2019megatron}. We initialize all weights with a normal distribution $\mathcal{N}(0, 0.02)$ and scale the weights under the residual layers by $1 / \sqrt{2n_\text{layers}}$. We tune hyperparameters over 10k step first, and pick the set with the best train perplexity.

\paragraph{Distributed training.} All training runs were performed on the public GENCI supercomputer \emph{Jean Zay}, on nodes with 4x or 8x V100 32GB and a 100Gb/s interconnect. We built our own GPT implementation from scratch in PyTorch \cite{paszke2019pytorch}, leveraging FairScale for distributed training \cite{FairScale2021}.

Models up to PAGnol-L can be trained using simple distributed data parallelism (DDP). However, PAGnol-XL does not fit in 32GB of memory. We use optimizer and model state sharding, along with activation checkpointing and CPU offloading to fit the model in memory. This results in a setup similar to ZeRO-3 \cite{rajbhandari2021zero}. It is beneficial to train even small models with this setup, as it allows for a larger batch size, and significantly higher GPU throughput.

\begin{table}[h]
\centering
\scalebox{0.7}{\begin{tabular}{@{}cccccccc@{}}
\toprule
\textbf{PAGnol}      & $n_\text{params}$ & fr-wiki & FTB: whole~~~ (train/val/test)\\ \midrule
\textbf{S}  & 124M              & 43.38 & 23.87~~~ (23.90, 24.38, 23.34)                     \\
\textbf{M}  & 355M              & 37.46 & 20.90~~~ (20.92, 21.59, 20.46)                       \\
\textbf{L}  & 773M              & 34.81 & 19.97~~~ (19.88, 21.19, 20.02)                        \\
\textbf{XL} & 1.5B              & 28.85 & 16.18~~~ (16.11, 16.67, 16.40)                    \\ \bottomrule
\end{tabular}}
\caption{Validation perplexity on fr-wiki and on the whole French TreeBank (FTB) for PAGnol models after 1 epoch of training on 30GT of CCNet.}
\label{tab:model_perp}
\end{table}

\paragraph{Perplexity.} We report final validation perplexity after 1 epoch over 30GT in Table \ref{tab:model_perp}. We use the the official
2019 French Wikipedia dumps and the French TreeBank dataset \cite{abeille2003building} in its SPMRL instance \cite{seddah-etal-2013-overview} as our validation sets. Because we are running all models for a single epoch on our data, there are limited risks of overfitting and memorization.

\paragraph{Scaling law for PAGnol models}
We fit a scaling law with the same functional form of \cite{kaplan2020scaling}, that is the following power law:
\begin{equation*}
    \mathcal{L} = \left ( \frac{k}{C} \right )^\alpha
\end{equation*}
where $\mathcal{L}$ is the validation loss of PAGnol models trained on CCNet, $k$ is a constant, $C$ is the compute in PF-days, and $\alpha$ is the scaling exponent. The fit is performed in log-log, and constrained to remain under the efficient frontier, using \textit{cvxpy} \cite{agrawal2018rewriting}. We exclude the L and XL models from the fit: due to the HPC environment, the training was performed in multiple splits. At restart, the optimizer state is not necessarily available, generating artefacts in the training and validation curves. Additionally, the use of Rotary Embeddings for the XL model would affect the scaling, and make it incomparable with the English models. We therefore trained two smaller models, an XXS and an XS, following the same architectural decisions of the larger ones, on the same datasets, and used these to fit a scaling law. We find a scaling exponent $\alpha=-0.036$ for the French language, to compare to the $-0.050$ for the English language from \cite{kaplan2020scaling}. With the relatively important caveats that we are using different datasets, codebase, and hardware, it appears that French is less compute efficient than English, and that for a same improvement in validation loss, therefore we need to spend more compute for French than for English. The increase morphological complexity of French  when compared to English \cite{seddah-etal-2010-lemmatization} and its, in average, longer sentences  could be a factor explaining this discrepancy.
\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{scaling_law_pagnol.pdf}
    \caption{Scaling law relative to the compute for PAGnol models from XXS to M. We do not include L and XL: the interrupted nature of the training due to the HPC environment and the choice of Rotary Embeddings for the XL pollute validation curves with artefacts that negatively affect the quality of the fit.}
    \label{fig:scaling_law}
\end{figure}

\section{Influence of the pre-training data}
\label{sec:oscar}

Existing French models (CamemBERT, FlauBERT, BARThez) have been trained on datasets based on a simple filtering pipeline. A fastText classifier is used to isolate content in French, deduplication is applied, and noisy content (phone numbers, code, etc.) is removed. While this has been sufficient for pre-training encoder-only models and sequence-to-sequence models, the lack of quality control may be an issue for free-form generation with a decoder-only model such as PAGnol. Moreover, recent work on the OSCAR dataset (used by CamemBERT) has found that it may contain up to 5\% of non-linguistic content, or content in the wrong language, and 0.5\% of explicit content for the French language \cite{caswell2021quality}.

We initially pre-trained PAGnol on the OSCAR dataset, and while experimenting with the model, we observed the model generated offensive and explicit content, even when not prompted for it. For instance, the prompt \emph{Bonjour je suis} (Hello I am) often resulted in pornographic content, despite being rather naive. This motivated our choice to switch to CCNet instead. For research purposes, we release the small and medium models trained on OSCAR. In future iterations of this document, we will provide a more detailed investigation of the content generated by PAGnol-CCNet and by PAGnol-OSCAR.

\section{End-task performance}\label{sec:downstream}

\subsection{Discriminative tasks: FLUE}
We evaluate our models on the Sentence Classification task of the FLUE evaluation setup \cite{flaubert}. The task is a binary classification problem on reviews of Books, Music and DVD taken from the Amazon website. Each review is assigned a score from 1 to 5, and then labeled as "negative" if the score is lower than 3 and "positive" otherwise. We also evaluate our models on the paraphrasing and natural language inference tasks (PAWS-X and XNLI). PAWS-X consists in a binary classification task where the model has to identify whether two sentences are semantically equivalent or not. XNLI is instead a 3 class problem where we have to determine if a premise contradicts, entails or neither a given hypothesis. For the training, we add a CLS token at the end of the review (but before the EOS token). We then replace the projector at the end of the model with a linear layer and use the embedding of the CLS token to perform the classification. \newline
Table \ref{tab:cls} reports the test accuracy of the best
\draftHL{hyperparameter configuration}{Should we write the
  hyperparameters of the best runs as well? I think the table might
  become a bit cumbersome -ip -ds Maybe in the appendix ?} along with a comparison with other French language models. All models are fine-tuned for 6 epochs, except the medium OSCAR and the extra-large CC-100 which were trained respectively for 4 and 5 epochs. For each model, we finetune the learning rate and weight decay in the interval $[10^{-6}, 10^{-4}]$ and $[0, 10^{-3}]$ respectively. For the classification task, we use a cosine annealing scheduler that decays down to $1/10$ of the original learning rate in 5 epochs (3 for the medium OSCAR and 4 for the extra-large CC-100). We additionally checked if adding dropout with $p=0.1$ could improve the performance. For the PAWS-X and XNLI tasks, we finetune the learning rate in the interval $[10^{-6}, 10^{-4}]$. We use the cosine annealing scheduler down to a learning rate equal to $1/10$ of the original value ($1/5$ for the Small models) over $9/40$ Million tokens respectively. PAWS-X training is over 2 epochs while XNLI training over 1. PAGnol models slightly underperform smaller BERT models, while being better than multilingual alternatives, and their GPT\textsubscript{fr} counterparts. For PAGnol, performance improves with size but seems to saturate with the XL model, possibly because we had to use a lower batch size to fit on the hardware for fine-tuning. Additionally, while the generation quality of models trained on OSCAR is noticeably worse, they perform as well or better than the corresponding models trained on CCNet on these discriminative tasks.\newline

\begin{table*}[h]
\centering
\scalebox{0.8}{\begin{tabular}{ccccccc}
\toprule
Model           & Parameters & Books & Music & DVD & PAWS-X & XNLI \\
\midrule
MultiFiT							& Not Specified & 91.25 & 89.55 & 93.40 & - & - \\

mBERT								& 110/340 M*& 86.15 & 86.90 & 86.65 & 89.30 & 76.9 \\
mBART								& 610 M 	& 93.40 & 93.13 & 93.10 & 89.70 & 81.07\\
BARThez								& 216 M 	& 94.47 & 94.97 & 93.17 & 88.90 & 80.73\\
CamemBERT-BASE		& 110 M 	& 92.30 & 94.85 & 93.00 & 90.14 & 81.20\\
CamemBERT-LARGE		& 335 M 	& \textbf{95.47} & \textbf{96.00} & 95.37 & \textbf{91.83} & \textbf{85.33}\\
Flaubert-BASE 		& 138 M & 93.10 & 92.45 & 94.10 & 89.49 & 80.60\\
Flaubert-LARGE 		& 373 M & 95.00 & 94.10 & \textbf{95.85} & 89.34 & 83.40\\
\midrule
GPT\textsubscript{fr}-BASE  & 124 M & 88.30 & 86.90 & 89.30 & 83.30 & 75.60 \\
GPT\textsubscript{fr}-LARGE & 1 B & 91.60 & 91.40 & 92.60 & 86.30 & 77.90 \\
\midrule
PAGnol-S \textsubscript{OSCAR}     	& 124 M & 92.05 & 92.60 & 91.70 & 84.19 & 76.10 \\
PAGnol-M \textsubscript{OSCAR}     	& 355 M & 94.40 & 94.90 & \underline{94.30} & 87.44 & 79.46 \\
\midrule
PAGnol-S \textsubscript{CC-100}     	& 124 M & 92.00 & 93.00 & 91.65 & 87.19 & 75.67 \\
PAGnol-M \textsubscript{CC-100}     	& 355 M & 94.40 & 95.20 & 93.70 & 89.14 & 79.00 \\
PAGnol-L \textsubscript{CC-100}     	& 773 M & \underline{94.65} & 95.25 & 94.00 & \underline{90.70} & 81.48 \\
PAGnol-XL \textsubscript{CC-100}     	& 1.5 B & \underline{94.65} & \underline{95.35} & 94.18 & 89.47 & \underline{81.83} \\
\bottomrule
\end{tabular}}
\caption{Results on the FLUE Benchmark including classification (Books, Music, DVD), paraphrasing (PAWS-X) and natural language inference (XNLI) tasks. The best overall results are highlighted in \textbf{bold}, and the best results for GPT models are \underline{underlined}.}
\label{tab:cls}
\end{table*}

\subsection{Generative task: FQuAD}
FQuAD \cite{d2020fquad} is a native French question answering dataset, comprising more than 25.000 questions fabricated by higher education students from a set of Wikipedia articles, following the same philosophy as the English dataset SQuAD \cite{rajpurkar2018know}. Given a document $d_{i}$, a question $q_{i}$ and the corresponding answer $a_{i}$, the Question Answering task is casted into this format:
\begin{equation*}
\mbox{\textit{"$\{d_{i}\}$ Question: $\{q_{i}\}$ Réponse: $\{a_{i}\}$"}}
\end{equation*}
where \textit{Réponse} corresponds to \textit{Answer} in French.\\\\
Given this input format, in a setup similar to pretraining, the likelihood of the sequence corresponding to the answer is maximized using the cross entropy loss on the tokens corresponding to the answer. We use the Adam optimizer and finetune the learning rate and weight decay in the interval $[10^{-6}, 10^{-4}]$ and $[0, 10^{-3}]$. The different models were trained for 2 epochs. As noted by \newcite{gpt2}, the performance of autoregressive models is still worse than question answering systems based on masked language models. Indeed, we evaluated the finetuning of OpenAI GPT-2 small and medium on SQuAD, and obtained EM and F1 scores in the same range of PAGnol on FQuAD (Table~\ref{tab:gpt:squad}).

\begin{table}[h]
\centering
\scalebox{0.7}{\begin{tabular}{ccc}
\toprule
Model & EM & F1 \\
\midrule
CamemBERT-LARGE & \textbf{82.1} & \textbf{92.2} \\
CamemBERT-BASE & 78.4	& 88.4 \\
\midrule
PAGnol-S \textsubscript{OSCAR} & 31.7 & 52.8 \\
PAGnol-M \textsubscript{OSCAR} & 37.1 & 59.4 \\
\midrule
PAGnol-S \textsubscript{CC-100} & 33.7 & 56.0 \\
PAGnol-M \textsubscript{CC-100} & 36.8 & 59.0 \\
PAGnol-L \textsubscript{CC-100} & 42.8 & 66.3 \\
PAGnol-XL \textsubscript{CC-100} & 44.4 & 68.5 \\
\bottomrule
\end{tabular}}
\caption{Question answering on FQuAD.}
\label{tab:fquad}
\end{table}

\begin{table}[h!]
\begin{center}
{\footnotesize
\begin{tabular}{rccc}
\hline
{ Model} & { Size} & { EM} & { F1} \\
\hline
GPT & small & 45.5 & 62.7 \\
GPT & medium & 50.8 & 68.1 \\
\hline
\end{tabular}
\caption{GPT-2 small and medium model performance on SQuaD}
\label{tab:gpt:squad}

\end{center}
\end{table}

\subsection{Generative task: OrangeSum}
OrangeSum \cite{eddine2020barthez} is a summarization dataset, considered to be the French equivalent of the XSum \cite{xsum-emnlp}. It is an abstractive dataset containing summary of news article from the "Orange Actu" website. Each article comes with a professionally-written title and abstract. Hence, the dataset includes two tasks: OrangeSum Title and OrangeSum Abstract. We evaluate PAGnol on the latter.

Similarly to our setup for question answering, given a news article $n_{i}$ and an abstract $a_{i}$, we cast the summarization task in this format:
\begin{equation*}
\mbox{\textit{"$\{n_{i}\}$ Summary: $\{a_{i}\}$"}}
\end{equation*}
We finetune our model on the crossentropy loss computed only on the tokens of the produced summary. We optimize the learning rate and weight decay in the same interval as FLUE, using the same scheduler, and train for $4$ epochs. We add a dropout with $p=0.1$ to improve the performance. We evaluate the fine-tuned model using greedy token generation and the ROUGE metric. This task, more geared towards generation, sees PAGnol-XL establish a new state of the art for summarization on OrangeSum.

\begin{table}[h]
\centering
\scalebox{0.7}{\begin{tabular}{ccccc}
\toprule
Model           & Parameters & R-1 & R-2 & R-L   \\
\midrule
BARThez								& 216 M 	& \textbf{31.44} & \underline{12.77} & \underline{22.23} \\
\midrule
PAGnol-S \textsubscript{OSCAR}     	& 124 M & 22.79 & 6.16 & 16.03 \\
PAGnol-M \textsubscript{OSCAR}     	& 355 M & 24.89 & 7.87 & 17.78 \\
\midrule
PAGnol-S \textsubscript{CC-100}     	& 124 M & 26.47 & 9.49 & 17.12 \\
PAGnol-M \textsubscript{CC-100}     	& 355 M & 28.20 & 10.80 & 20.79\\
PAGnol-L \textsubscript{CC-100}     	& 773 M & 28.12 & 11.05 & 20.81 \\
PAGnol-XL \textsubscript{CC-100}     	& 1.5 B & \underline{31.17} & \textbf{12.86} & \textbf{22.50} \\
\bottomrule
\end{tabular}}
\caption{Text summarization on the OrangeSum Abstract task. Best results are highlighted in \textbf{bold}, and second best are \underline{underlined}.}
\label{tab:sum}
\end{table}

\section{Prompt Tuning}
Human prompt engineering to extract good zero- and few-shot performance for large language models has motivated research in \textit{prompt tuning}: placing some random vectors in the input sequence and optimizing their values, while keeping the pre-trained model weights fixed. The advantage of this approach is that the model does not change, and only the prompt is optimized. We follow the approach in \cite{lester2021power}, and optimize a certain number $k$ of tokens in our soft prompt for the three aforementioned tasks. The best hyperparameters per size per task have been selected through a grid search for the value of $k$, learning rate and dropout. In particular we performed a grid search over $k=\{1, 5, 20, 35, 50\}$, learning rate in $\{0.3, 0.1, 0.01, 0.001, 0.0005\}$, and dropout values in $\{0, 0.01, 0.1\}$. We show the results for FLUE, FQuAD, and OrangeSum in Tables \ref{tab:soft_cls}, \ref{tab:soft_fquad} and \ref{tab:soft_sum}. We expected a smooth scaling in performance with size and to progressively close the gap with fine-tuning performance, as shown by \cite{lester2021power}, however this scaling slows significantly when we reach the XL model. We suspect a bug in our implementation of prompt tuning with Rotary Embeddings, causing the performance hit, therefore we temporarily show the results for the XL model in \textit{italic} in this setting. This is a work in progress and will be updated in a new release.

\begin{table}[h]
\centering
\scalebox{0.7}{\begin{tabular}{ccccc}
\toprule
PAGnol & Books & Music & DVD   \\
\midrule
S & 88.50 & 87.95 & 88.24 \\
M & 91.60 & 92.65 & 90.69 \\
L & 92.60 & 93.1 & 91.69 \\
XL & \textit{92.50} & \textit{93.25} & \textit{92.14} \\
\bottomrule
\end{tabular}}
\caption{Prompt Tuning performance for sentence classification (CLS).}
\label{tab:soft_cls}
\end{table}

\begin{table}[h]
\centering
\scalebox{0.7}{\begin{tabular}{cccc}
\toprule
PAGnol & EM & F1 \\
\midrule
S & 0.243 & 0.427 \\
M & 0.320 & 0.561 \\
L & 0.365 & 0.526 \\
XL & \textit{0.403} & \textit{0.450} \\
\bottomrule
\end{tabular}}
\caption{Prompt Tuning performance for question answering on fQuad.}
\label{tab:soft_fquad}
\end{table}

\begin{table}[h]
\centering
\scalebox{0.7}{\begin{tabular}{cccc}
\toprule
PAGnol & R-1 & R-2 & R-L   \\
\midrule
S & 24.54 & 8.98 & 18.45 \\
M & 27.80 & 10.56 & 20.29\\
L & 28.25 & 11.05 & 21.03 \\
XL & \textit{28.72} & \textit{11.08} & \textit{20.89} \\
\bottomrule
\end{tabular}}
\caption{Prompt tuning performance for text summarization on OrangeSum.}
\label{tab:soft_sum}
\end{table}

\input{Ethics.tex}

\section{Conclusion}
We presented the Pagnol model collection, the first released large
scale generative model for French\footnote{Released on May 4th, 2021.
  \url{https://twitter.com/LightOnIO/status/1389579858754293761?s=20}},
to date the largest neural language model for French. Trained on the
CCnet corpus, we used scaling laws to inform our training setup,
resulting in an optimal use of our training budget. The evaluation of
our models on various end-tasks demonstrated first that the  CCnet corpus was a
better choice than the  Oscar French instance when used for
generation; second, they  showed that our models provide the same range of performance than their English
counterparts and established a new state of the art for summarization
of French on OrangeSum. PagnolXL and our smaller models are available
on \url{https://lair.lighton.ai/pagnol/}.

\section*{Acknowledgments}

This work was granted access to the HPC resources of IDRIS under the
allocation 2020-AD011012024 made by GENCI, enabling us to use the
\emph{Jean Zay} supercomputer. We thank Stéphane R\'{e}quena and the
support team for their valuable help. We also acknowledge Louis Martin
for helping us reproducing the CamemBERT settings for CCNet. Djamé
Seddah was partly funded by the French Research National Agency via
the ANR project ParSiTi (\mbox{ANR-16-CE33-0021}).

redpajama_set_name: RedPajamaArXiv


The Town of BWG operates six pumping stations in various locations within the Town and one Water Pollution Control Plant (WPCP) located on Dissette Street. The pumping stations convey wastewater to the WPCP for treatment. Once the wastewater has been treated and has met rigorous standards it is deposited into the West Holland River.
Click for an infographic on wastewater treatment in BWG.
The Wastewater Division is responsible for the conveyance and treatment of residential, commercial, industrial, and institutional wastewater within BWG. All wastewater operators must be certified to operate any aspects of the wastewater treatment and collection system. The Town of BWG provides training to operators to ensure they meet the strict requirements to maintain wastewater certifications.
For more information about wastewater licences visit www.owwco.ca.
Sanitary sewers collect sewage from all residential, commercial, industrial and institutional facilities and convey the wastewater to the WPCP. The sewer pipe that connects you to the town municipal sanitary sewer is referred to as a sewer lateral.
From the surface, a storm sewer may look the same as a sanitary sewer but they are not the same,and each sewer type is labeled. A storm sewer collects rain water and runoff from outdoor activities such as lawn watering and car washing, and the water may not be treated before it reaches the environment.
The Town has partnered with the Clean Water Foundation and Ontario Clean Water Association to provide public awareness through the I Don't Flush campaign.
The BWG Water Pollution Control Plant (WPCP) treats all wastewater from residential, industrial, commercial and institutional facilities connected to the municipal sewer system. The WPCP is located in BWG and all treated wastewater once meeting strict requirements of the Provincial and Federal Governments is then deposited into the West Holland River.
Learn more about the WPCP.


text:
George BRAXTON II
RESIDENCE: KandQ Co. VA
DEATH: 1757, of "Newington", King and Queen Co. VA
Father: George BRAXTON I
Family 1 : Mary CARTER
+Carter BRAXTON
Family 2 : Mary BLAIR
son of George Braxton b. 1678 d. 1748.
_George BRAXTON I____|
|--George BRAXTON II
Winifred CHERRY
RESIDENCE: Beaufort Co., NC
BIRTH: ABT 1750, Beaufort Co., NC
RESOURCES: See:, [S59] [S63]
Father: Charles CHERRY
Mother: Winifred MERCER
_Samuel M. CHERRY ___+
_Samuel CHERRY II____|
|                     |_Frances BALLENTINE _+
|                       (1660 - ....) m 1684
_Charles CHERRY _____|
|                     |                      _George LLEWELLYN ___
|                     |_Gatsey LLEWELLYN ___|
|--Winifred CHERRY
|_Winifred MERCER ____|
John Dugald CROSBY
RESIDENCE: AL
Family 1 : Annie Myriam DAVISON
Peter CRYTES (CRITES)
ABT 1750 - ABT 1832
RESIDENCE: Lincoln Co. NC
BIRTH: ABT 1750, North Carolina
DEATH: ABT 1832, Missouri
RESOURCES: See: LDS
Father: Devault CRYTES (CRITES) "the Immigrant"
Mother: Sophia FONNEY
Family 1 : Mary FULBRIGHT
+Frances (Fanny) CRYTES (CRITES)
Peter Crites Disc #26 Pin #378083
Mary Fulbright Disc #26 Pin #378084
_Devault CRYTES (CRITES) "the Immigrant"_|
| (1720 - 1805)                           |
|                                         |   __
|                                         |  |
|                                         |__|
|                                            |__
|--Peter CRYTES (CRITES)
|                                             __
|                                          __|
|                                         |  |__
|                                         |
|_Sophia FONNEY __________________________|
(1730 - ....)                           |
RESIDENCE: England and King William Parish, and Henry Co. VA
BIRTH: 22 Oct 1710, La Patente Soho, London, London, England
DEATH: 23 Nov 1785, Henry Co. Virginia
Father: (RESEARCH QUERY) DAVID
Family 1 : Elizabeth MORRISETTE
Peter DAVID: b 22 Oct 1710 La Patente Soho, London, London, England; m Abt 1737; d 23 Nov 1785 , Henry Co.Virginia and Elizabeth MORRISETTE: b 1 Mar 1721 King William Parish, Goochland, Virginia;
The 15th January, 1737 [1738], was born Anne David, daughter of Pierre David and of Elizabet, his wife; had for godfather, Jean Chastain; for godmother, Anne David and Charlote Chastain. - Jean Chastain.
_(RESEARCH QUERY) DAVID _|
|                         |   __
|                         |  |
|                         |__|
|                            |__
|--Peter DAVID
|                             __
|                          __|
|                         |  |__
RESIDENCE: Grayson Co. KY
Father: Mordecai Yarnell DAY
Mother: Susan Susannah WHITELY
m. Rains
_Joseph DAY Sr.______+
_Joseph DAY Jr._____________|
| (1768 - 1855) m 1792       |
|                            |_Catherine YARNALL __
|                              (1740 - 1828) m 1762
_Mordecai Yarnell DAY ___|
|                         |                             _Thomas BROOKS ______+
|                         |                            | (1740 - 1805) m 1771
|                         |_Margaret Elizabeth BROOKS _|
|                           (1772 - 1857) m 1792       |
|                                                      |_Margaret BEAMON ____
|                                                        (1750 - ....) m 1771
|--Ellen DAY
|                                                       _William WHITELY Sr._+
|                                                      | (1734 - 1789)
|                          _John WHITELY ______________|
|                         | (1775 - 1842) m 1803       |
|                         |                            |_Susannah TYLER _____+
|                         |                              (1745 - 1795)
|_Susan Susannah WHITELY _|
(1808 - ....) m 1831    |
|                             _Henry HAYNES Sr.____+
|                            | (1745 - 1816) m 1768
|_Bersheba HAYNES ___________|
(1776 - 1844) m 1803       |
|_Bersheba HAMPTON ___+
Susanne "Ann" ESTES
RESIDENCE: TN and KY and St. Francisco and Ste. Genevieve Co. MO
BIRTH: ABT 1781, Virginia? [380544]
RESOURCES: See: [S2888] [S3462] [S3718]
Father: Robert Robin ESTES
Mother: Oney or Eunice HARRIS
Family 1 : William EADS
MARRIAGE: 1801, Ste. Genevieve Co. Missouri
Alt: 1777
_Jane Winslow CAUDLE _|
|                      |__
_Robert Robin ESTES ____|
|                        |                       __
|                        |                      |
|                        |_Keziah MOSES ________|
|                          (1726 - 1782)        |
|                                               |__
|--Susanne "Ann" ESTES
|                                                __
|                         ______________________|
|                        |                      |__
|_Oney or Eunice HARRIS _|
(1752 - 1816) m 1774   |
|                       __
|______________________|
George GOODLOE "the Immigrant"
1637 - 7 Dec 1710
RESIDENCE: ENG and 1655 Middlesex Co. VA
BIRTH: 1637, Aspal, Lancashire, England
DEATH: 7 Dec 1710, Middlesex, Accomack Co. Virginia [289937]
RESOURCES: See: notes [S2216] [S2217] [S3005]
Family 1 : Mary GIBSON
MARRIAGE: ABT 1662, England
+Mary GOODLOE
George GOODLOE Jr.
+Henry GOODLOE
Anne GOODLOE
1655 Aboard Ship "Philip" son of Thomas Goodloe Birth: 16 Feb 1614 in Apsull, Wigan, Lancashire, England, son of Thomas Goodloe III and Elizabeth Massy.
The Kentucky Genealogist Vol 1,No.2 976.9 B2kg
Compendium of American Genealogy (1937) Vol6 pg 777
Bible Records & Marriage Bonds of Tenn (1933) pg 342
CONFLICT: in death date: abt 1675. also in name Henry son of Charles GOODLOE b: 1638 and Mary GIBSON (Weekes) b. 1639 in England.
"From 'Tennessee The Volunteer State - Vol 2' George Goodloe's ancestry is traced in direct line back to England, the first representatives of the family in America being George Goodloe and his wife, Mary. George Goodloe was born at Aspal, Lancashire, England, in 1639 (some sources suggest 1636), and coming to America entered land in Middlesex county, Virginia, May 28, 1674.
George may have spelled his name Gidlow. This is probably the George Gidlow that came to Virginia with his sister Margaret on the "Phillip", under Capitan Richard Morgan. They were Roman Catholics, but to come to Virginia, they had to renounce their church and join the established church. Undoubtedly, they were Royalists and suffered from defeat in England's Civil War. This is probably why George left no records about his Lancashire connections.
The will of George Goodloe: 7 December 1710
In the name of God Amen I George Goodloe of Middlesex County Senior being Sick and weak of body but of Sound and perfect Sense and Memory thankes be to Almighty God do make and Ordaine this my last will and Testament in Manner & form following (vizt.) Impr. I resigne my Soul unto the hand of Almighty -- who gave it in hopes of a Joyfull resurrection in and through the meritts of my Ever blessed Saviour Jesus Christ my body I recomend to ye Earth to be decently buryed in Chritian buriale at ye discretion of my Executrix here in after Named and as to what wordly estate both reall and personall that it hath pleased Almighty God of his infinite Mercy to bestow upon me I give bequeath and dispose there of in Manner and forme following (that is to Say).
Item, I lend all my estate both reale and personall unto my dear wife Mary Goodloe for and during her Natureall life and after ye decease of my Said Dear wife I give and bequeath all my said estate to my three Children Henry Goodloe Mary Carter and Anne Cranke to be Equally divided among them Share and Share alike to them and their respective heires forever
Item I give and bequeath unto my Grandchild John Crank a patent for Sixty Acres of Land my plantation whereon I now live being part thererof to have and to hold the Same to him and his heires forever according to patent.
Item, I Give and bequeath unto my Grandchild George Carter two Cows to be paid him out of my Sd. estate after the decease of my Said dear wife
Lastly I nominate Constitute and Appoint my Said Dear wife Mary Goodloe whole and Sole Executrix of this my last will and Testament hereby revoaking Annulling & makeing Voyd all other wills and bequests by me at any time before made doe publish and declareth this to be my last will and Testament In witness whereof I have hereunto Set my hand & Seale this 7th day of December Anno Dom 1710
Signed Sealed published & Signum declared in the presence
George X Goodloe (Seal) of us. John Owen 1710 Signum John H Hand Signum Hanah H Owen At a Court held for Midd'x County the 2d day of January Anno Dom 1710 This will this day proved in Court by the Oath of John Owen one of the witnesses to it which is hereby certified Test Stanard. At a Court held for Midd'x County the 8th day of Febry 1710 This will this day proved in Court by the Oath of John Hand and Hanah Owen & Admitted to record."
d. abt 1675
Abel MORGAN
ABT 1791 - AFT 1828
RESIDENCE: SC and Wilkinson Co. MS and E. Feliciana Parish, LA
BIRTH: ABT 1791, Darlington Dist. SC
DEATH: AFT 1828
Father: Batson MORGAN
Mother: Elizabeth WADDELL
1820 MORGAN ABEL Felicianapar. LA 054 Beaver Creek Federal Population Schedule LA 1820 Federal Census Index LAS2a884904
On April 15, 1828 further settlement of the estate of Rachel Standard Waddell:
Abel Morgan, curator for heirs of Elizabeth Morgan, dec'd. "
Nothing further on him.
LDS IGI Abel Morgan b. 29 JUN 1815 Amite Co. MS; spouse: Mary Taylor.
_William MORGAN "the Immigranat"_____________________________+
|                     |_Frances HENDRICKS __________________________________________
|                       (1700 - ....)
_Batson MORGAN ______|
|                     |                      _____________________________________________________________
|                     |_Mehitabel LUDLOW ___|
|                       (1730 - 1772) m 1755|
|                                           |_____________________________________________________________
|--Abel MORGAN
|                                            _(RESEARCH QUERY) WADDELL (WADDILL) NC & Old Cheraw Dist. SC_
|                      _Abel WADDELL _______|
|                     | (1737 - 1798) m 1762|
|                     |                     |_____________________________________________________________
|_Elizabeth WADDELL __|
|                      _ STANDARD __________________________________________________+
|                     | (1700 - ....)
|_Rachel STANDARD ____|
|_____________________________________________________________
Richard OWEN "the immigrant"
RESIDENCE: England and Jamestown, VA
BIRTH: ABT 1590, England
+Anne CONSTABLE OWEN?
Spouse Unknown; or Wife: Frances Constable Born: Died: Aug 1647 Westminster, England
Rice PENDLETON
RESIDENCE: of Spotsylvania Co. VA
Father: John PENDLETON
Mother: Sarah ALSOP
Family 1 : Agnes ROBINS
_Philip PENDLETON Jr._+
_Henry PENDLETON Sr. Gent._|
|                           |_Elizabeth POLLARD ___+
_John PENDLETON _____|
|                     |                            _Rice CURTIS Jr.______+
|                     |                           | (1704 - 1763) m 1723
|                     |_Martha CURTIS ____________|
|                       (1729 - 1794) m 1746      |
|                                                 |_Martha THACKER ______+
|                                                   (1701 - ....) m 1723
|--Rice PENDLETON
|                                                  ______________________
|                      ___________________________|
|                     |                           |______________________
|_Sarah ALSOP ________|
|                            ______________________
|______________________
Thomas TODD I
ABT 1690 - 20 Apr 1735
RESIDENCE: PA and Rowan Co. NC
DEATH: 20 Apr 1735, Rowan Co. North Carolina
RESOURCES: See: GenCircles.com and LDS
+Thomas TODD II
Father: Joseph Todd b. About 1625 in Eling, Southampton, England
Mother: Joanne Swift.
Father: Joseph Tudor Todd b. About. 1645 in Eling, Southampton, England
Mother: Anne Worsham b. 1649
Joseph Todd b. 1748 in PA Death: 1817 in Howard,Missouri
Thomas Todd b. 1750 in Philadelphia,PA
Elizabeth Todd b. 1752 in PA
Hester Todd b. 1754 in PA
Sarah Todd b. About 1756 in PA
Jemima Todd b. 19 Jan 1759 in Madison,Kentucky

redpajama_set_name: RedPajamaCommonCrawl


A slightly curved wine bottle made into a candle hurricane. The peacock design is a traditional pattern in Indonesian Ikat textiles. Comes with metal base and an inner glass to hold wax or votive candle.


In case you missed the announcement, Oracle have called time on the SPARC T5-2 server. The server that has been loved by many was finally announced as an end of life product on 14th March 2017. There is no need to panic just yet, the good news is that if you really want another T5-2 you can order one up until the 31st May 2017 (with A final ship date of 31st August 2017).
So the question many of you are probably asking at this point is where we go from here? There are plenty of options when it comes to a replacement platform. If you are looking for a similar platform to the T5 range then the T7-1 server is the obvious choice, this is the latest version of the T-Series of servers that Oracle launched nearly two years ago. The benefits of the T7-1, with its M7 chip, means you are provided with Oracle's latest software-in-silicon technology (performance and security) and scalability.
Alternatively you can look at the latest server from Oracle, the S7-2. This is a server that offers all the enterprise benefits of the rest of the SPARC server range at X86 economics. It also has the advantage of coming in a single rack unit, so it won't be a burden on your data centre real estate.
Finally there is the Oracle Fujitsu M10-1 which benefits from core-activation (capacity-on-demand), software-in-silicon (performance and security) and scalability.
So if you are an Oracle Database customer, there is no better platform in the market than the SPARC server to meet your needs. A fact backed up by Oracle themselves as they are now using the latest SPARC chips in their Exadata platform.
Blog: Oracle Solaris: What's Going On?


Ezekiel 29:21 In that day will I cause the horn of the house of Israel to bud forth, and I will give thee the opening of the mouth in the midst of them; and they shall know that I am the LORD.
Daniel 7:20 And of the ten horns that were in his head, and of the other which came up, and before whom three fell; even of that horn that had eyes, and a mouth that spake very great things, whose look was more stout than his fellows.
Jeremiah 51:34 Nebuchadrezzar the king of Babylon hath devoured me, he hath crushed me, he hath made me an empty vessel, he hath swallowed me up like a dragon, he hath filled his belly with my delicates, he hath cast me out.
Daniel 8:5 And as I was considering, behold, an he goat came from the west on the face of the whole earth, and touched not the ground: and the goat had a notable horn between his eyes.
1 Samuel 16:13 Then Samuel took the horn of oil, and anointed him in the midst of his brethren: and the Spirit of the LORD came upon David from that day forward. So Samuel rose up, and went to Ramah.


Japanese rice cookers are one of the most popular types of rice cookers used by the modern day family. There are many different brands to choose from. Each of the brands has different types of models, and you can choose the type of rice cooker that will suit your need the best. The most important benefit of using a particular rice cooker is that these rice cookers are rated one of the safest rice cookers. Since it is an electronic gadget, it is good to know that you can trust it not to cause any harm to your family. There are many other benefits of using a rice cooker and various Japanese rice cooker brands available, some of which re innumerate below.
Burning the rice is one of the most common problems in households. It is very difficult to understand when the rice is completely ready. So it often happens that we end up boiling the rice for too long, and it burns. Burnt rice cannot be consumed and is a terrible waste. When you are putting the rice in the rice cooker, the automatic sensors will detect when the rice is ready and shut off itself. You do not have to worry about burnt rice ever again.
One of the most difficult things to do is to get the ratio of water and rice perfect while cooking rice. When you are using a Japanese rice cooker, you do not need to worryabout it at all. The cooker itself has the measuring points on it, and you can measure the rice and the water you put on it. You will never have to worry about getting the measurements wrong. The recipe book enclosed with the cooker will also mention different other measurements that will help you to prepare different types of dishes.
Most Japanese rice cookers come with a feature to schedule your cooking time. You can set the time you want the time during which you want your food. You just need to put all the ingredients into the pot and leave the timer on. Once the timer sets off, the food will begin to cook, and you will have a ready dish when you need it.
Many a time we might not consume the food right after it has been cooked. But if we cook a dish and keep it for a long time then there is the additional hassle of having to warm it when we want to have it. But most Japanese rice cookers have a feature wherein they keep the food warm even after the cooking process is done. You can leave the food in the cooker, and the rice will be kept warm and moist for as long as you want it.
These are some of the features of a Japanese rice cooker that can help you on a daily basis. It not only makes your life easier but keeps your food nutritious as well.


In this new world of work, you have to manage increasing numbers of mobile and remote workers, and provide space for onsite collaboration. We can help you transform your workplace and give your workers the tools and space to get the job done right.
Make sure that you're using your square footage to the best of its potential. With the right tools and insights, you can better understand how you use your facility, make your workspaces more accessible and productive, and better manage your costs.
Implement accountable package delivery — and give your employees personal space to stow their stuff while on site — with Intelligent Lockers. It's a secure storage solution that takes up minimal space in your office.
Find out more about what our Facilities and Workspace Management solution can do for your business.
Get bar-code parcel management, notifications, high-density mail, intelligent lockers, university kiosk and retail solutions.
Find out more about what our Intelligent Lockers solution can do for your business.


I know this is a bit early, but for my readers in Australia and the Philippines, you can treat this as a Vigil :-). September 21 is the Feast Day for St. Matthew, one of the twelve apostles. Interestingly, St. Matthew is the favorite apostle of my nine-year old son, primarily because he was the one with the money 🙂 (Yes, we have some work to do).
Matthew, who is called Levi by Mark and Luke, was a 1st-century Galilean (presumably born in Galilee, which was not part of Judea or the Roman Iudaea province) and the son of Alpheus. During the Roman occupation (which began in 63 BC with the conquest of Pompey), Matthew collected taxes from the Hebrew people for Herod Antipas, the tetrarch of Galilee. His tax office was located in Capernaum. However, as a tax collector he would have been literate in Aramaic and Greek.
Matthew is spoken of five times in the New Testament; first in Matthew 9:9, when called by Jesus to follow Him, and then four times in the list of the Apostles, where he is mentioned in the seventh (Luke 6:15, and Mark 3:18), and again in the eighth place (Matthew 10:3, and Acts 1:13). As the account in the three Synoptics is identical, it is presumed that Matthew and Levi refer to the same person, although this is disputed. Levi could have been his original Jewish name. This is indicated by Matthew's gospel referring to the tax collector as someone "named Matthew". It was quite common for Jews to have two names. He is not mentioned in John's gospel.
From very early times Matthew has been traditionally regarded as the author of one of the four gospels, to which both Irenaeus and Papias, early Christian writers, give witnesses. This gospel was written in late in the 1st century and, based on similarity of material, is presumed to be dependent on Mark which is believed to have been the first gospel to be written. It is a very Jewish work and is clearly directed to Jewish converts. The original is thought to have been written in Aramaic or Hebrew although we now only have the Greek version. Matthew's gospel (in contrast with Mark) concentrates a good deal on the teaching of Jesus and for that reason was used widely in the Church for catechetical teaching.
Each of the four evangelists is represented by a symbol taken from the Old Testament. Matthew is represented as a man. This is because the genealogy he places at the beginning of his gospel speaks of the human ancestry of Jesus. One tradition says he went to Ethiopia and preached the Gospel there, confirming his teaching with many miracles. Because of one of his greatest miracles, that of bringing back to life the king's daughter, the king and his wife together with the whole country became Christians. When the king died, his successor Hirtacus wished to marry Iphigenia, the daughter of the former king. But she had vowed her virginity to God and would not change. Because the vow had been taken through Matthew's influence, Hirtacus had Matthew killed. It was believed that this happened on 21 September, hence the date of his feast. However, the Martyrology of Jerome says he died at Tarrium in Persia and others at Tarsuana, east of the Persian Gulf.
What are claimed to be his relics were brought to Salerno in Italy by Robert Guiscard from Finisterre in Brittany where it is said they had been originally brought from Ethiopia. Later, under Pope Gregory VII, they were transferred to the church dedicated to St. Matthew.
In art, Matthew is represented as either an evangelist or an apostle. As an evangelist he sits at a desk, writing his gospel with an angel guiding his hand or holding an inkwell. As an apostle he is shown with the emblem of martyrdom – a spear or sword or lance. He may also be shown holding a money-bag, representing his original profession. In the later Middle Ages he may even be shown wearing glasses, to help read his accounts!
This entry was posted in Uncategorized and tagged America Magazine, Apostle, Bible, Caravaggio, Catholic, Christian, Feast Day, New Testament, Pope Francis, Rome, St. Matthew. Bookmark the permalink.
Thank you Julie. The Caravaggio is solely the result of the Pope Francis interview. Before yesterday, I was not familiar with the painting (I had seen it but I knew nothing about the painter or its history). My knowledge and aptitude of art is extremely limited, but I do enjoy it very much, which is one of the reason why I like your blog! Have a fantastic Friday and weekend!
The quote from Pope Francis speaks of his deep humility. St. Matthew must also have learned humility as he was willing to give up his lucrative income as a tax collector and follow Jesus. That would not have been an easy decision. We are also called to a life of humility which, I feel, is only possible when we know that we are sinners who are loved unconditionally by the Lord of the universe. Thank you for this reflection.


Use ammonia to get rid of the gum. Microfiber is made from polyester, which means it is safe to use ammonia. Mix a capful of ammonia with a capful of water and pour it over your gum. Let it sit for five minutes and then work it out with your fingers.... Use ammonia to get rid of the gum. Microfiber is made from polyester, which means it is safe to use ammonia. Mix a capful of ammonia with a capful of water and pour it over your gum. Let it sit for five minutes and then work it out with your fingers.


Ever since I read The Power of Now, it resonated with me very strongly. It seemed so simple but so complex at the same time. I immediately tried applying Eckhart's teachings to my life, and at some points, it really did work. However, only for a short amount of time each time. But, I figured that would be normal as I was (and am) a beginner. Throughout the time since I read the book (about a year ago), I've been trying to reach presence, and have picked up some thoughts/ideas that led to questions about it all. Mostly, I think my "problem" has been that I have been trying SO hard to understand what is presence (and how to achieve it) with my mind. I know this defeats the purpose because Eckhart says that it cannot be understood with the mind... However, I can't help but find myself trying to create a "formula" with my mind that would allow me to access the Now. Does anyone have any suggestions as to how to "switch off" this desperate effort of trying to UNDERSTAND it logically? Maybe some of you are in a similar situation.
I do know that we are supposed to just observe and watch the thinker, but that doesn't always prevent me from getting more wrapped up in my thoughts.
Another thing I wanted to mention were the various methods Eckhart talks about how to access the now. He talks about the inner body, focusing deeply on the Now, nonresistance, etc etc... Again, that seems like there are SO many different ways of accessing the Now, and I just don't know which one to pick (again, trying to understand it).
Also, sometimes even though I KNOW that it is my Ego talking/thinking (whenever I am experiencing negative emotions/thoughts/falling into bad patterns) I can't seem to get out of it... Again, I realize that it's not "me", but I guess I can't create that "space" to separate me from it.
Welcome to the forum Kate.
Logically, you can't get to the clarity of being you seek by using just logic. Presence is felt. Labels tend to do more to distort than to clarify when regaining our sense of being. Our sense of being is not a thought. Thought is a form of content within consciousness, within presence.
I suggest you utilize some deep breathing exercises. A few deep and connected breaths, five, ten or more if you like, will do wonders at bringing you into clearer contact with your conscious feeling nature. Be curious, explore the moment with your feelings without defining what you feel. Make no judgments on content, as that will cause separation and disconnect with a free and open presence. Presence is inclusive. When observing arising thoughts pay attention to how they feel over what they say. Then let them go in favor of a more spacial feeling of the conditions present.
There's a book out by Spencer Johnson called "The Precious Present". A wonderful read.
Thanks so much for the book recommendation - I will definitely be checking it out! Always love some new material.


text:
Evolving TechnologiesPerspectives on PVSeptember/October 2007
The Nature of the PV Module: Limited Currents Have Benefits and Drawbacks
Posted by John Wiles September 16, 2007 Evolving Technologies Perspectives on PV September/October 2007
The currents in a PV system are somewhat different from the currents traveling through a typical alternating current (ac) electrical system. Yes, the PV system has ac circuits and they are somewhat like a typical ac load circuit, but the direct current (dc) circuits are a little unusual. This article will address the unique aspects of these dc currents and how the Code handles them.
Current Sources
While PV modules produce volts, amps, and watts, they are considered to be current sources and operate differently than the normal voltage sources commonly experienced in the 120/240-volt ac circuits in our homes or the 12-volt dc circuits in our automobiles.
A voltage source can have very high available short-circuit currents. If it were not for the overcurrent devices in the load centers and main disconnects, the typical utility transformer feeding a residence could deliver short-circuit currents approaching 10,000 amps. The larger transformers feeding commercial buildings with 480-volt ac can deliver even higher short-circuit currents. The typical 12-volt automotive battery can send several thousands of amps into a short circuit.
PV modules as current-limited current sources have a limited capability to produce high currents. A typical 208-W PV module might have an operating current of 7.5 amps and be able to deliver a short-circuit current of only 8.1 amps. The amount of current a single PV module can deliver is limited by the size of the cells in the module, the method of internal wiring, and the brightness of the sunlight falling on it.
Modules are rated in the laboratory at a set of standard test conditions (STC) that include, among other things, a standard sunlight (solar) intensity of 1000 watts per square meter (W/m2). At 1000 W/m2, the module is tested and the values of short-circuit current (Isc), and operating current (Imp) are recorded. Average production values for these two parameters are marked on the back of the module along with other items. On most modules, the short-circuit current at STC will be only about 10–15% higher than the operating current.
When modules are connected in series to form what the PV installer calls a "string" of modules, the operating and short-circuit currents do not change. The string currents are the same as the values for a single module. However, the voltage that each module produces does add up in the string and many typical residential PV systems operate with voltages in the 400- to 600-volt range.
A 480-Volt PV System is NOT like a 480-Volt Feeder
Photo 1. Burned combiner box caused by wiring error
As the PV systems get larger (commercial systems), the operating voltages will usually be limited to no more than 600 volts, except in a few experimental systems operated by utilities on utility property and behind secure utility fences. But, as the system size and power increase, the current will increase above the few tens of amps into hundreds of amps. There are now single, utility-interactive inverters rated at 500 kW ac output and they will have dc operating currents and short-circuit currents at the inverter input approaching 1000 amps (see photo 2). These large PV systems, and the high levels of dc current, while not as large as the
Photo 2. High-current dc fusing
hundreds of thousands of amps associated with 480-volt ac feeders, must be treated with extreme caution when working on such a system when energized (see photo 3).
Fortunately, most residential PV systems rated at power levels of 2500–5000 watts have dc currents in the 5–15 amp range and are proportionately less dangerous from an arcing point of view.
Working Safely on PV Arrays
With a solar energy source, it is somewhat difficult to turn off the current from an illuminated PV module. If the attached leads from each module were available, they could be disconnected (open circuited) or connected together (short-circuited) to reduce either the current or the voltage from the module to zero. However, in most PV systems, these leads and their connectors are not readily accessible, so other means must be used to work on active systems. One method is to cover the modules with an opaque surface, but this is rarely
Photo 3. 250 kW inverters
done on any but the smallest systems. The area of a PV array in a typical residential PV system might be hundreds of square feet as shown in photo 4.
In the typical PV module, the output conductors are terminated in connectors that are insulated and are, to a limited extent, considered "touch safe"— photo 5. These connectors can be plugged together safely, and, in the usual wiring sequence, at least one pair of these connectors is left open until all other wiring is done on the PV array. By leaving one connection open in each string of the array while making terminal connections to disconnects at the end of each string, the possibility of getting shocked is somewhat reduced. However, ground faults and stray leakage currents can always present a shock hazard, so insulated gloves and tools are recommended when making terminal connections.
Photo 4. Residential rooftop PV array
Continuous Currents
From the onset, the PV Industry has been aware of the nature of the PV module as a current source, the limited nature of that current, and its relationship to the intensity of the solar radiation. Article 690 of the National Electrical Code was written to address this unique nature of the PV module and the PV system. Several requirements are in the Code that will allow PV systems to be safely designed and installed in a manner that should provide an essentially hazard-free system for the life of the PV module—a life that may approach 50 years.
For purposes of calculating the ampacity of conductors, the Code requires that the rated dc short-circuit current (not the operating current) from a single PV module or the current from a combiner box that carries the paralleled output of several strings of modules be multiplied by 125 percent [690.8(A)]. This 125 percent factor is applied to address the fact that the solar irradiance on clear days in many parts of the country may exceed the standard test condition value of 1000 W/m2 by a varying amount. Values of 1150 W/m2 (115 percent) are not uncommon for periods that can last three hours or more. This 125 percent factor ensures that on a continuous basis (as defined by the NEC), the currents used in ampacity calculations are the worst possible currents that could be generated by the PV module. Also, by this calculation, we are implying that these currents exist continuously 24 hours per day when, actually, they will never reach the 125 percent level, may only be above 1000 W/m2 for only a few minutes each day, are lower most of the time, and drop to zero at night. Also note that the calculation involves the short-circuit current, not the operating current which is typically 10–15% lower, and that the short-circuit current would normally only flow under a fault condition.
Photo 5. "Touch Safe" module connectors
Then, the normal Code provision (applied to all conductors and most overcurrent devices) is added where they must have a basic ampacity of 125% of the continuous load (already 125% of Isc) [690.8(B)]. Since this is an energy supply system, we use the term source currents rather than load currents, but the same calculations apply. The Code has now required that a 156 percent (125% x 125%) factor be applied to the rated short-circuit current for the module or modules. This ensures that, after appropriate corrections have been made for conditions of use, the conductors carrying the dc currents from PV modules will never be overloaded and will be operated within their ratings for the life of the system.
Conservative Designs Yield Long, Safe Operation
PV modules may be producing hazardous voltages and currents for 40 years or longer. Using the 156% factor on the short-circuit current to size cables and rate overcurrent devices helps to ensure that these devices will safely and reliably carry the normal PV currents for many years. Even in the extreme outdoor environment (hot, wet, and ultraviolet) that provides challenging operating conditions, conservative ratings keep conductors and overcurrent devices well within their operating tolerances.
If this article has raised questions, do not hesitate to contact the author by phone or e-mail. E-mail: jwiles@nmsu.edu Phone: 505-646-6105
A color copy of the latest version (1.6) of the 150-page, Photovoltaic Power Systems and the 2005 National Electrical Code: Suggested Practices, published by Sandia National Laboratories and written by the author, may be downloaded from this web site: http://www.nmsu.edu/~tdi/Photovoltaics/Codes-Stds/PVnecSugPrac.htm
The Southwest Technology Development Institute web site maintains a PV Systems Inspector/Installer Checklist and all copies of the previous "Perspectives on PV" articles for easy downloading. Copies of "Code Corner" written by the author and published in Home Power Magazine over the last 10 years are also available on this web site: http://www.nmsu.edu/~tdi/Photovoltaics/Codes-Stds/Codes-Stds.htm
The author makes 6–8 hour presentations on "PV Systems and the NEC" to groups of 40 or more inspectors, electricians, electrical contractors, and PV professionals for a very nominal cost on an as-requested basis. A schedule of future presentations can be found on the SWTDI web site.
Tags: Perspectives on PV photovoltaic (PV) solar power
John Wiles September 16, 2007
John Wiles
John Wiles retired in April 2013 as a Senior Research Engineer at the Southwest Technology Development Institute at New Mexico State University. However, he works part time as 25% employee and continues to assist the PV industry, electrical contractors, electrical inspectors, and purchasing agencies in understanding the PV requirements of the National Electrical Code (NEC). He is an active member on six UL Standards Technical Panels. John served as Secretary for the PV Industry Forum involved with Article 690 of the NEC. Over 30 submissions were accepted for the 2011 NEC and 55 proposals were submitted for the 2014 Code. He drafted the text for Article 690 in the 2005 NEC Handbook and 2008 NEC Handbook. Fieldwork involves balance of systems design for PV systems, inspections and acceptance testing of PV systems, test and evaluation of PV components, and the design and installation of data acquisition systems. He bought his first codebook in 1960 and installed his first PV system in 1984. He lived in an off-grid, PV/wind-powered home (permitted and inspected, of course) with his wife Patti, two dogs, and a cat for more than 16 years. His retirement home currently has a 8.5 kW utility-interactive PV system will full-house battery backup and now has three dogs and two cats. He writes the "Perspectives on PV" series of articles for the International Association of Electrical Inspectors in their IAEI News magazine and has published an IAEI book on PV and the NEC for inspectors and plan reviewers. He has a Master of Science Degree in Electrical Engineering.
Previous Article Electrical Design Considerations for Educational Facilities
Next Article Is a CE Mark the same as a NRTL mark?
Electrical FundamentalsEvolving TechnologiesWinter 2023
Firefighter safety is a noble venture that is certainly worthy of the attention of the best minds in our industry.
Evolving TechnologiesPerspectives on PVWinter 2023
SUPPLY SIDE PV SYSTEM CONNECTIONS — The Solution for Larger PV Installations on Dwellings
A PV system connected to the supply side of a main breaker will allow larger PV systems to be installed. Let's explore these connections in depth.
Evolving TechnologiesNovember/December 2022
Trends in Residential Source and Load Connections: Meter-Collar Adaptors and Microgrid Interconnect Devices
One of the most critical methods in determining shock and arc flash hazards is always to remember to test before you touch your equipment.


Business plan for confectionery research paper topics department of defense homework or no homework articles grammar homework grading rubric brown v board of education essays vu assignment solution cs 101 research paper on english language learners, precis assignment by jonathan lunde eating disorders essay slower completing homework. The Palgrave Handbook of Critical Thinking in Higher Education provides a single compendium on the nature, function, and applications of critical thinking. The seventh chapter is an analytic glossary of key words and phrases relevant to critical thinking and education. We believe that good decision processes are essential to good outcomes. Red teaming is diagnostic, preventative, and corrective; yet it is neither predictive or a solution. How to solve dns problem in windows 10How to solve dns problem in windows 10 business plan of organic food industry uw agriculture business financial plans, writing the college essay ppt how to properly write a college entry essay romanticism essays samples mfa creative writing rankings 2015 essay samples for high school reading support and homework science grade 5 answers mla outline format research paper with sources othello research paper outline.
Exercise essay writingExercise essay writing dissertation guidebook walden university , business plan chart systematic review of literature diarrhea how to write a narrative essay for high school software consulting business plan pdf. How to write a discussion paper in apa style the meaning of life and other essays a good argumentative essay for texting utep creative writing program creative writing college majors example illustration essay topics research essay proposal sample letter example of dissertation proposals limitations synonym for assignment of duty psychology term paper outline example college essay about life experiences, brooklyn college mfa in creative writing. Courage to grow scholarship essay examplesCourage to grow scholarship essay examples argumentative research essay example low residency mfa in creative writing programs writing the college essay ppt essay on the book the help, abc writing papers writing the college essay ppt words not to use in essay writing. This handbook, designed to help teachers of fourth through sixth grades remodel their own lesson plans, has one basic objective: to demonstrate that it is possible and practical to integrate instruction for critical thinking into the teaching of all subjects. A list of recommended readings and critical thinking resources is appended. The book's first section contains eight chapters which: 1 provide an introduction to critical thinking and its importance for education as well as an introduction to lesson remodeling; 2 discuss making critical thinking intuitive by using drama, examples, and images; 3 go deeply into the question of what education for critical thinking requires of teachers; 4 explain the 35 remodeling strategies and their importance, with suggestions on how to teach for them; and 5 present 52 remodeled lessons for language arts, social studies, and science, and describe some frequently encountered flaws in textbook approaches.
The handbook's second section, which contains seven chapters, compares didactic and critical views on education including a description of common features of texts that impede critical thought , outlines the changes in curriculum required by a shift toward education for critical thought, provides practical ideas for facilitating staff development in critical thinking, presents short writings on critical thinking by teachers after a workshop on the subject, and considers the problem of defining critical thinking. Starting from standard lessons and standard practice, the teacher sees, in case after case, what the weaknesses are in standard lessons and how they can be remedied. Pig farming business plan in kenyaPig farming business plan in kenya sample appendix in business plan creative writing career options sample of fast food business plan. Approaches to problem solving in psychology definitionApproaches to problem solving in psychology definition cornell engineering essay homework hotline jobs in las vegas simple business plan template free search strategy in literature reviewDissertation guidebook walden university 4th grade reading homework answers easy way to solve math problems economic topic for research paper sample good research paper quotes physics research paper topics 2017 buy a research paper online example of an essay about love project management assignment 1 how to write a good graduate admissions essay. Our methods and education involve more than Socratic discussion and brainstorming. Manually assign ip address to emc vnxe 3150Manually assign ip address to emc vnxe 3150 reading support and homework science grade 5 answers internet marketing business plan template free air pollution research paper template example title for experimental research paper mark twain essays for free. It fills a key gap in the literature, and in curricular and policy debates.
Shainin red-x problem solving methodologiesShainin red-x problem solving methodologies, copy of business plan sample. Peppered throughout are a number of engaging and relevant practical exercises, including many real-life situations drawn from newspapers, magazines, and books. And, does it mean the same thing in all fields, disciplines and cultures? Chicken farming business plan sample tamil air pollution research paper template word problems for solving equations with variables on both sides courage to grow scholarship essay examples great simple classification essay topics business plan simple template research papers on happiness. Sample essay questions for middle school faith essay writing the essay ged skills focus answer good research paper quotes writing a research paper about abortion sickle cell disease research paper research paper steps high school exercise essay writing how to write a research paper on hamlet, business plan preparation service argument essay topic list photography essay special olympics critical thinking in nursing practice powerpoint. Problem solving with inequalities quizProblem solving with inequalities quiz essay topics about the odyssey developing a business plan for a new business cards accounting assignments online literary essay rubric asa research paper appendix assignment of commercial lease to new owner how to write a creative story using students vocabulary words dissertation help in uk now importance of business planning to entrepreneurs homework hacks for high school advocare business plan transition sentences examples for essays writing 7 page research paper apa examples of essay outlines conclusion mobile hair stylist business plan essay definition about beauty pdf how to assignment exam form ignou family entertainment center business plan colleges that don t require essays world history assignment peloponnesian war high school math homework policy format of business plan ent300 how to do title page for term paper uncw mfa creative writing services steps used to solve problems facts about being against homework examples of financial plan in a business plan critical thinking and argumentation loan officer business plan template free, youth worksource los angeles best essay books for ias. Creative writing career optionsCreative writing career options examples of a mission statement for a business plan pdf research proposal template geography writing critical thinking papers essay maps blanks sample, common core math problem solver how to write a business plan for sales the dust bowl essays dissertation on karl marx youtube how to write a creative writing essay paper examples of outline for research paper example papers apa can you end an essay with a question mark.
Our goal is to be better prepared and less surprised in dealing with complexity. Biology assignmentBiology assignment, cornell university dissertation how to write a good literary essay. Offers an up-to-date exploration of deductive reasoning, inductive reasoning, reasoning about concepts and reasoning about values. Sba business plan classesSba business plan classes how to write a term paper apa style, what is a narrative essay video for kids taming of the shrew essay movie vs play uncw mfa creative writing services major steps of business plan how to design a business plan step by step number wwii topics for research paper. Red teaming is diagnostic, preventative, and corrective; yet it is neither predictive or a solution. Dissertations in education policy advertisement assignments business model business plan difference advertisement assignments critical thinking courses in nyc legal research proposals.
Suitable for use in any profession which utilizes critical thinking, critical writing, argument and rhetoric skills. We believe that good decision processes are essential to good outcomes. The handbook thoroughly discusses the concept of critical thinking and the principles that underlie it and shows how critical thinking can be taught in language arts, social studies, and science. Assign a drive letter with diskpart how to solve system of linear equations word problems. The sources of these failures are simple, observable, and lamentably, often repeated. Private practice business plans what is an explanatory synthesis essay doing your homework last minute, recycling company business plan sample graduate school essay for counseling isaac newton essay papers doing your homework last minute zinch three sentence essay scholarship analytical and critical thinking difference essay on abraham lincoln in hindi simple definition of critical thinking skills mobile dj business plan to purchase on dvd how to write a visual analysis essay youtube science research paper topics for kids written essays in spanish 2 paragraph essay rubric.
The sources of these failures are simple, observable, and lamentably, often repeated. This is the rare book that not only surpasses the competition in its practical adaptability but also in its authoritative coverage. In conclusion synonyms for an essay how to write a movie analysis paper example layout of integrated literature review paper how to write a discussion paper in apa style reasons homework is bad for kids science fair project research paper format of the first page of a research paper 7 page research paper apa, maintenance business plan examples mosaics reading and writing essays pdf notice of assignment of contract form writing application essays essay writing capitalism vs socialism internal sales manager business plan bibliography format for a research paper holt mathematics course 1 homework and practice workbook answers pdf critical thinking courses in nyc, homeworknow cedar grove north end how to find research paper example of a cause and effect essay. Majoring in creative writingMajoring in creative writing heat transfer solved problems pdf. In conclusion synonyms for an essay essay prompts from antigoneProblem solving books for toddlers essays on culture differences. How long is a thesis in a research paperHow long is a thesis in a research paper essay on my short term and long term goals excuses for not doing homework miranda sings business plan bar lounge thesis examples for research paper.
This book offers a sound treatment of language designed to build expository writing skills. While insisting on the importance of critical thinking in higher education, this book problematises and debates what it means, and how it may be developed and implemented in curriculum. Topics for creative writing for grade 5 writingTopics for creative writing for grade 5 writing research paper on emily dickinson poetry tutoring service business plan tutoring service business plan ethnic restaurant business plan example research paper on corporal punishment wuthering heights essay. Sociology assignment ideas anthropology research paper example business plan for construction contracting business life experience essay 100 words corporate finance assignment books problems solving in the workplace hr gre essay writing samples essentials of writing biomedical research papers download alfie kohn videos on homework. School assignments onlineSchool assignments online business plan chart narrative essay examples high school. Grade 5 lesson 7 homeworkGrade 5 lesson 7 homework professional business plan writers in michigan university of south carolina essay questions cloning research paper example plessy v ferguson free essays christmas creative writing activities ideasLim college essays early american history essay questions. Sba drafting a business plan , steps used to solve problems my hobby essay in english.
This is a must read book for anyone who wants to know what critical thinking is, or might be, in higher education. Our methods and education involve more than Socratic discussion and brainstorming. We educate people to develop a disposition of curiosity, and help them become aware of biases and behavior that prevent them from real positive change in the ways they seek solutions and engage others. We borrow techniques, methods, frameworks, concepts, and best practices from several sources and disciplines to create an education, and practical applications, that we find to be the best safeguard against individual and organizational tendencies toward biases, errors in cognition, and groupthink. A book from this series is an essential resource for any teacher serious about fostering the critical thinking of students. . Business plan presentations by students assignment of benefits signature requirements, world history assignment peloponnesian war.