text:
\section{Overview}
The main intuition behind the Text-DIAE framework has been the degradation invariant features learnt during pre-training without any manually annotated samples, which is fine-tuned for key downstream application tasks: Text-Recognition and Document Image Enhancement. In this supplementary material, we discuss and highlight some essential insights of Text-DIAE under the two sections~\ref{s:recognition} and ~\ref{s:enhancement}.

\section{Text Recognition}\label{s:recognition}
\subsection{Transformations}
We begin showing the augmentations employed to the input images used at the pre-training stage. We carefully select transformations suited for text in order to not disrupt the sequential information of characters found at the word level.

Results of transformations employed can be seen in Figure~\ref{fig:trans_da}. The transformations used are: addition of Gaussian noise, shear, minor rotation, scaling and cropping upon some detected points given by a mask in order to not alter the characters in a significant manner.
Code regarding the data augmentation pipeline employed will be made public at \url{UponAcceptance}.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{./images/supp_transformations.pdf}
\caption{\textbf{Transformations during training.} Image augmentations are carefully selected in order to not disrupt the sequentially nature of characters contained in a word.}
\label{fig:trans_da}
\end{figure}

\setlength{\tabcolsep}{2pt}
\begin{table*}[t]
\begin{center}

\caption{\textbf{Implementation Details.} Implementations details of Handwritten and Scene Text Recognition. The acronyms STR and HTR stands for Scene Text Recognition and Handwritten Text Recognition respectively.}
\label{tab:model_configs}
\resizebox{\textwidth}{!}{
\begin{tabular}{llll}
\hline\noalign{\smallskip}
ConFigure & Pre-Training & Fine tuning & Scratch \\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
optimizer & AdamW & Adam & Adam \\
learning rate & 1.5 $e^{-4}$ & 1 $e^{-4}$ & 1.5 $e^{-5}$ \\
weight decay & 0.05 & 0.05 & 0.05 \\
optimizer momentum & $\beta_1$, $\beta_2$=0.9, 0.95 & $\beta_1$, $\beta_2$=0.9, 0.95 & $\beta_1$, $\beta_2$=0.9, 0.95 \\
batch size & 64 (HTR) / 192 (STR) & 64 (HTR) / 256 STR & 64 (HTR) / 256 STR \\
learning rate schedule & cosine decay & cosine decay & cosine decay \\
warmup epochs & 3 & 3 & 10 \\
training epochs & 100 (HTR) / 2 (STR) & 600 (HTR) / 10 (STR) & 600 (HTR) / 10 (STR) \\
\hline
\end{tabular}}
\end{center}
\vspace{-0.5cm}
\end{table*}
\setlength{\tabcolsep}{1.4pt}

\begin{figure*}[h]
\centering
\includegraphics[width=\linewidth]{./images/supp_htr_qualitative.pdf}
\caption{\textbf{Reconstruction of Handwritten Text.} Qualitative samples of reconstructed input images. The model may have seen the word, but it has not seen the images at training. The reconstruction is performed after employing masking as pretext task during pre-training.}
\label{fig:htr_masking}
\end{figure*}

\subsection{Datasets}
In this work, we make use of the following public datasets to train models in handwriting and scene text recognition.
\begin{itemize}
    \item \textbf{IAM}~\cite{marti2002iam}: It is a handwritten English text dataset. It contains 657 different writers, and it is partitioned into writer independent training, validation and test. Addtionally, this dataset is comprised of 74,805 fully segmented words.
    \item \textbf{CVL}~\cite{kleber2013cvl}: Dataset of handwritten text in English language. It was written
    310 different writers, partitioned into writer independent training and test sets. In total, 27 writers wrote 7 texts and the remaining 283 writers wrote 5 texts.
    \item \textbf{IIIT5K-words} (IIIT5K)~\cite{mishra2012scene}: This is a dataset made of 2000 training and 3000 testing cropped scene text images from the Internet. Often employed in recognition and scene text retrieval tasks.
    \item \textbf{ICDAR-2013} (IC13)~\cite{karatzas2013icdar}: It is a dataset that contains 848 training and 1015 testing cropped scene text in focused environments, where the text was relevant. Often employed in text detection and recognition.
    \item \textbf{Modified-SynthText}: This dataset is a modified version of~\cite{gupta2016synthetic}. It was employed by~\cite{gomez2018single} to train a scene-text retrieval model. It contains 4M cropped scene text images which were generated synthetically. This dataset is smaller than the original one and it avoids random flipping of text images and rotated text that can break the sequential order of characters.

\end{itemize}
The pre-training datasets are domain-dependant, in handwritten text we used IAM and CVL. In scene text we used the Modified-SynthText as well as  the training sets from ICDAR13~\cite{karatzas2013icdar} and IIIT5K~\cite{mishra2012scene}.
\subsection{Evaluation metrics}

\noindent
\textbf{Character Error Rate (CER).} CER is used to compare the similarity of two strings at character level. Given two stings $s_1$ and $s_2$, where $s_2$ is the reference (ground truth):
\begin{equation}
    CER(s_1,s_2) = \frac{S+D+I}{N}
\end{equation}
where $S$ is the number of substitutions, $D$ is the number of deletions and $I$ is the number of insertions that are applied in $s_1$ to obtain $s_1 = s_2$. Noting that,  $N$ is the number of characters in $s_2$.

\noindent
\textbf{Edit Distance at 1 (ED1).} It represents the accuracy of a prediction up to a edit distance of 1. This implies that a prediction is correct compared to the ground truth if only \textit{one} substitution, deletion or insertion is required for both strings to match.

\subsection{Implementation details}

In order to provide the reader with reproducibility of the presented work, the implementation details of the models showcased for Handwritten and Scene Text Recognition are shown and summarized in Table~\ref{tab:model_configs}.

During pre-training we deploy an encoder with 6 layers and 8 attention heads to encode the input, with a dimension of 768. We used this same number of layers and attention heads for the decoder, with a dimension of 512 in text recognition and 768 for document enhancement.
At masking, each input image (with size $64\times256\times3$ for text recognition and $256\times256\times3$ for document enhancement) is divided into a set of patches with size $8\times8\times3$. Similarly as~\cite{he2021masked}, we employ random masking of 75\% of the patches. To add blur, we add average blur with random kernel sizes between 1 and 15. In order to add background noise, we add weighted contrasting backgrounds from different text documents.  The pre-training was done for 2 epochs in scene text and 100 epochs for the handwritten domain.

In the fine tuning stage, we use the same encoder (pre-trained) with a different decoder of 6 layers, 8 attention heads and dimension of 768.

\subsection{Qualitative Insights}

\begin{figure*}[h]
\centering
\includegraphics[width=\linewidth]{./images/supp_str_qualitative.pdf}
\caption{\textbf{Reconstruction of Scene Text.} Qualitative samples of reconstructed input images. The model may have seen the word, but it has not seen the images at training. The reconstruction is performed after employing masking as pretext task during pre-training.}
\label{fig:str_masking}
\end{figure*}

\begin{figure*}[h]
\centering
\includegraphics[ width=\linewidth]{images/reco_qualitative_htr.pdf}
\caption{\textbf{Handwritten Text Recognition.} Qualitative samples of attaching a decoder and fine-tuning an encoder to perform recognition of IAM (black font) and CVL (blue font). Correctly predicted samples are displayed in green, while mistakes are shown in red. Failure cases are depicted on the last row.}
\label{fig:text_rec_qualitative}
\end{figure*}

In Figure~\ref{fig:htr_masking} we show qualitative samples of the original, masked and reconstructed images at pre-training. The images correspond to the validation set and the model have not seen them. In Figure~\ref{fig:htr_masking}, we focus on the pre-text task (masking - 75\%) that yields the biggest improvement on the downstream task. We note that at pre-training the model learns a distribution of characters (English) that helps to solve the masking task. It is worth noting the capability of keeping similar font types and applying a padding in short words. In harder cases, such as in the words "stucked", "saying" and "having" the model still reconstructs blurry or wrong characters. However, we have to note the complexity of this task, which is very demanding even for humans.

Similarly, we show qualitative samples of reconstructing images at pre-training by masking an input scene-text cropped word in Figure~\ref{fig:str_masking}. The model has not seen this images before. The model learns to properly reconstruct input images, while properly keeping the background color and specific font style. Some failure cases are shown while reconstructing "pea" and "earthsellers".

We show in Figure~\ref{fig:text_rec_qualitative} the recognized text in the test sets of IAM and CVL datasets. Despite the difficulty of some samples, the model achieves state-of-the-art in IAM. Failure cases depict problematic handwritten styles to be recognized, e.g. "tonight" and "Christmas". Some characters are mistakenly predicted, thus causing errors in the next sequence of characters, such in "irresolute" and "wiener". Some errors come from difficult cropped samples such as "strikes", in which the sample starts with a strikethrough word.

\section{Document Image Enhancement}\label{s:enhancement}

\subsection{Datasets}

\begin{itemize}
    \item \textbf{DIBCO and H-DIBCO datasets}: These datasets were introduced in the Document Image Binarization challenges that took place since 2009. In this work we have mainly chosen two DIBCO(2011~\cite{Pratikakis2011icdar} and 2017~\cite{pratikakis2017icdar2017}) and two H-DIBCO(2012~\cite{pratikakis2012icfhr} and 2018~\cite{Pratikakis2018icfhr}) benchmarks for final validation on the document image binarization task. All the image samples are mainly historical documents that has undergone several distortions like bleed, show-through, smears, fading and so on. The DIBCO 2011 and 2017 contained 16 and 20 samples respectively. While the H-DIBCO 2012 and 2018 contained 14 and 10 instances respectively.

    \item \textbf{Blurry Document Images}: The training dataset contains 4000 training images and 932 images for validation of resolution 300x300 patches. This data was used in ~\cite{souibgui2020gan} and it is originally a subset from the dataset proposed in~\cite{hradivs2015convolutional}. Every instance is extracted from a different document image and each blur kernel used is exclusive.

\end{itemize}
 For pre-training we used 203,576 unlabeled document image samples. These samples were taken from the historical document DIBCO benchmarks (2009, 2010, 2013, 2014, 2015, 2016)~\cite{pratikakis2016icfhr2016}, Publaynet~\cite{zhong2019publaynet}, Palm-leaf~\cite{burie2016icfhr2016} and IAM Handwriting database.
\subsection{Evaluation metrics}

\noindent
\textbf{Peak signal-to-noise ratio (PSNR).} PSNR helps to establish a pixel-wise validation and measures the effectiveness of document image enhancement approaches in terms of visual quality. It computes the ratio between the maximum possible value of the signal(image) to the amount of noise (distortion) that affects the quality.  The higher the value, the more similar are the two images which in turn assures that the reconstructed image (binarized or deblurred) has a better quality. MAX is the maximum possible pixel value of the image (eg. MAX is 255 for pixels represented as 8 bits per sample) . Given two $M*N$ images, they can be fomulated as shown in eqn.~\ref{eq:mse_psnr}.

\begin{equation}
\begin{split}
    M S E=\frac{\sum_{M, N}\left[I_{1}(m, n)-I_{2}(m, n)\right]^{2}}{M * N}
    \\ P S N R=10 \log _{10}\left(\frac{MAX^{2}}{M S E}\right)
\end{split}
\label{eq:mse_psnr}
\end{equation}
\\

\noindent
\textbf{F-Measure(FM).} The F-measure gives the harmonic mean between the precision (P) and recall (R) scores. Precision determines the number of positive predictions, while recall measures the ability to find the positive predictions in a binary classifier. Here, in binarization problem, FM computes the accurate prediction of the white background and the black foreground (text) between a binarized sample and the ground-truth(GT) sample. FM is formulated as shown in eqn.~\ref{eq:fmeasure}.

\begin{equation}
\begin{split}
   F M=\frac{2 * \text {Recall} * \text {Precision}}{\text {Recall}+\text {Precision}}
\end{split}
\label{eq:fmeasure}
\end{equation}
\\

\noindent
\textbf{Pseudo F-Measure ($F_{ps}$).} The Pseudo F-Measure was first introduced for document image binarization task in~\cite{ntirogiannis2012performance} as it proposed a normalization procedure on the weights of the GT image(both background and foreground) for a lower penalization.
\\

\noindent
\textbf{Distance Reciprocal Distortion (DRD).} DRD was mainly used to measure visual distortion for all pixels in binary document images as proposed in~\cite{lu2004distance}. It is computed at first kth flipped pixel using a normalized weight matrix (5x5) $W_{norm}$ defined in the same paper~\cite{lu2004distance}. The formulation is done as shown in eqn.~\ref{eq:drd}. To calculate the overall DRD score between the binarized resultant image $B_{k}(x, y)$ and the GT image $G T_{k}(i, j)$  , it is basically to sum up the DRD for $N$ number of flipped pixels starting from $k=1$ and $NB_{nu}$ refers to the number of non-uniform blocks as formulated in eqn.~\ref{eq:drd_final}.

\begin{equation}
\begin{split}
   D R D_{k}=\sum_{i=-2}^{2} \sum_{j=-2}^{2}\left|G T_{k}(i, j)-B_{k}(x, y)\right| \times W_{norm}(i, j)
\end{split}
\label{eq:drd}
\end{equation}

\begin{equation}
    \begin{split}
        D R D=\frac{\sum_{k=1}^{N} D R D_{k}}{NB_{nu}}
    \end{split}
    \label{eq:drd_final}
\end{equation}

\subsection{Implementation details}

The implementation details of the models showcased for Document Image Enhancement tasks (binarization and deblurring) are shown and summarized in Table~\ref{tab:model_configs_enhancement}. The Xavier-Glorot uniform~\cite{glorot2010understanding} was used to initialize the transformer blocks, as implemented in the original ViT~\cite{dosovitskiy2020image}. The ViT-Base model variant was used as the main model baseline with the patch size $8$ and Random resized Crop was used as the default applied transformation for all the images during pre-training.
The pre-training was done for 100 epochs.
\setlength{\tabcolsep}{2pt}
\begin{table}[h]
\centering
\caption{\textbf{Training settings:} Text-DIAE model configurations and hyperparameters adapted for Document Image Enhancement tasks.}
\label{tab:model_configs_enhancement}
\begin{tabular}{lcc}
\hline
ConFigure                & \multicolumn{1}{l}{Pre-Training} & \multicolumn{1}{l}{Fine tuning} \\ \hline
optimizer              & AdamW                            & AdamW                           \\
learning rate          & 1.5e-4                           & 1.5e-4                          \\
weight decay           & 0.05                             & 0.05                            \\
optimizer momentum     & (0.9,0.95)                       & (0.9,0.99)                      \\
batch size             & 64                               & 64                              \\
learning rate schedule & cosine                           & cosine                          \\
warmup epochs          & 3                                & 15                              \\
training epochs        & 50                               & 100                             \\ \hline
\end{tabular}
\end{table}
\setlength{\tabcolsep}{1.4pt}

\subsection{Qualitative Insights}
The visual appearance of a historical document has been negatively affected with time by different forms of degradation. Recovering the document with its foreground text and information is the key challenge in document image enhancement. In this section, we will discuss some qualitative highlights of our proposed Text-DIAE model applied to document image binarization. We shall also cover some analysis on the task of document deblurring mainly on camera-captured document images affected by motion, out-of-focus blurs etc.

\begin{figure*}[h]
\centering
\setlength{\tabcolsep}{0.31em}
\begin{tabular}{c c c c}
     \toprule
     \scriptsize{Original Input} & \scriptsize{DocEnTr~\cite{souibgui2022docentr}} & \scriptsize{\textbf{Ours}} & \scriptsize{Ground Truth} \\
     \midrule
     \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/another_orig.png} &  \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/another_docentr.png} & \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/another_pred.png} &
     \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/another_gt.png} \\
     \midrule
     \begin{tikzpicture}
  \node (ocr1) [text width=0.21\textwidth, font=\fontsize{7pt}{7pt}\selectfont,
  align=left] at (0,0) {
  \textbf{OCR output:} ``{\fontfamily{qcr}\selectfont
    \color{Red}{J}\color{Green}{anythingob}\color{Red}{f}\color{Green}{cure,\\notunder}\color{Red}{f}\color{Green}{tood}\color{Red}{,,., ... ,}\color{Green}{Be here, the State expounds the darke}\color{Red}{f}\color{Green}{t wood: And makes the thicke}\color{red}{f}\color{Green}{t thickets p}\color{Red}{!}\color{Green}{ain}\color{Red}{-}\color{Green}{and clear,}}''
 };
\end{tikzpicture} &  \begin{tikzpicture}
  \node (ocr1) [text width=0.21\textwidth, font=\fontsize{7pt}{7pt}\selectfont,
  align=left] at (0,0) {%
  \textbf{OCR output:} ``{\fontfamily{qcr}\selectfont
  \color{Red}{I}\color{Green}{any thing ob}\color{Red}{f}\color{Green}{cure,notunder}\\ \color{Red}{f}\color{Green}{tood Be here, the State expounds the darke}\color{Red}{f}\color{Green}{t wood: A}\color{Red}{o}\color{Green}{d makes}\color{Red}{c}\color{Green}{he thicke}\color{red}{f}\color{Green}{t thickets p}\color{Red}{!t}\color{Green}{in}\color{Green}{and clear,}}''
 };

\end{tikzpicture} & \begin{tikzpicture}
  \node (ocr1) [text width=0.21\textwidth, font=\fontsize{7pt}{7pt}\selectfont,
  align=left] at (0,0) {%
  \textbf{OCR output:} ``{\fontfamily{qcr}\selectfont
  \color{Green}{I}\color{Red}{j}\color{Green}{any thing ob}\color{Red}{f}\color{Green}{cure,not under}\color{Red}{f}\color{Green}{tood Be here, the State expounds the darke}\color{Red}{f}\color{Green}{t wood: And makes}\color{Red}{c}\color{Green}{he thicke}\color{red}{f}\color{Green}{t thickets pl}\color{Red}{t}\color{Green}{in}\color{Green}{and clear,}}''
 };
\end{tikzpicture} &
     \begin{tikzpicture}
  \node (ocr1) [text width=0.21\textwidth, font=\fontsize{7pt}{7pt}\selectfont,
  align=left] at (0,0) {%
  \textbf{OCR output:} ``{\fontfamily{qcr}\selectfont
  \color{Green}{F}\color{Green}{any thing ob}\color{Red}{f}\color{Green}{cure,not under}\color{Red}{f}\color{Green}{tood Be here, the State expounds the darke}\color{Red}{f}\color{Green}{t wood: And makes}\color{Red}{c}\color{Green}{he thicke}\color{red}{f}\color{Green}{t thickets p}\color{Red}{!}\color{Green}{ain}\color{Green}{and clear,}}''
 };
\end{tikzpicture} \\
     \midrule
     \scriptsize{CER: 17.97} &  \scriptsize{CER: 13.28} & \scriptsize{CER: \textbf{11.50}} & \scriptsize{CER: 10.94} \\
     \bottomrule
     \end{tabular}
\caption{\textbf{Qualitative analysis of binarized samples for OCR.} The document image on the left refers to the originally captured image, followed by the DocEnTr~\cite{souibgui2022docentr} binarized result, and the Text-DIAE binarized result, and the ground-truth image in the last column. The correctly predicted OCR output is shown in "Green" font while the inaccurate ones are depicted in "Red" and recognition performance in terms of CER.}
\label{fig:ocr_dibco}
\end{figure*}

\noindent
\textbf{Text-DIAE recovers show-through.} The show-through problem appears when ink impressions from one side of the paper start appearing on the other side, making the document almost illegible. As shown in the results of examples (2nd, 3rd, 4th, 5th and 6th) of Figure~\ref{fig:our_results_dibcos_details} Text-DIAE binarized results get rid of the show-through successfully to make the document easily readable. An other qualitative comparison with other approaches (refer to Figure~\ref{fig:binarization_qualitative_2}) reveals the superiority of our approach than the related work in this task.
\\

\noindent
\textbf{Text-DIAE recovers smears or stains.} As shown in the 1st example of Figure~\ref{fig:our_results_dibcos_details} documents also suffer from stains or smears and need to be recovered for proper readability. Text-DIAE not only successfully recovers the text (foreground) but also smoothens its boundary pixels.
\\

\noindent
\textbf{Text-DIAE recovers faint characters or weak text.} Every individual glyph (character) inside a document can either appear faded due to the ink quality or paint as they start shrinking with time. Even adaptive handcrafted binarization approaches fail to recover and extract the text accurately. We illustrate in Figure~\ref{fig:ocr_dibco} an example of thin parchment of text and run the Tesseract-OCR engine (with their already in-built adaptive binarization tool) for validating the Text-DIAE model. The predicted OCR output from the original input image is compared with the binarized results from Text-DIAE, DocEnTr and the ground-truth image. The improvement in CER when compared with the original input and DocEnTr proves that Text-DIAE works substantially well to recover weak text for an OCR to read. Also, in the 8th example of Figure~\ref{fig:our_results_dibcos_details}, Text-DIAE performs quite good for thin characters under a faded background.
\\

\noindent
\textbf{Text-DIAE recovers contrast variations.} Most of the times in historical degraded documents, they suffer from huge differences between high/low pixels due to occlusion, illumination and other noisy factors. In the examples shown in Figure~\ref{fig:our_results_dibcos_details} the binarized results have been obtained under several different contrasting background variations. This can be attributed to the feature representations learnt during pre-training stage by Text-DIAE by applying background noise transformation.
\\

\noindent
\textbf{Text-DIAE excels in deblurring task.} When it comes to camera-captured documents, it often suffers from different kinds of blurring. Motion blur artefacts occur due to the relative speed between the object and the camera, where a sudden camera movement leads to a degradation in the captured image. The out-of-focus blur mainly occur when light fails to converge in the during camera capture. In this work, Text-DIAE was mainly applied to the benchmark data proposed in~\cite{hradivs2015convolutional} to see how the model performs in document deblurring. The results shown in Figure~\ref{fig:our_results_deblur_details} exhibit how the deblurred results from Text-DIAE achieves almost ground-truth level recovery from both motion-blur and out-of-focus blurring effects. The enhancement of the blurred images also help to capture the text from the deblurred sample, an illustration of which has been already presented in the main section of the paper. Also, the deblurring learning objective which is applied during the pre-training stage contributes significantly to improve the performance on the blurred document samples.
\\

\noindent
\textbf{Robustness Evaluation of Text-DIAE.} We use the same model pre-trained on the DIBCO datasets (2009, 2010, 2013, 2014, 2015, 2016), Palm Leaf images~\cite{burie2016icfhr2016} and IAM~\cite{marti2002iam} handwriting database and use it on fine-tuning for two different document image enhancement tasks: binarization and deblurring, without any special change in settings. This proves the robustness of Text-DIAE as it learns universal feature representations from different kinds of degradation (background noise, blur and masking) beneficial for multiple downstream tasks.

\begin{figure*}[!t]
\centering

 \begin{tabular}{ccc}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/4_2011.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/4_pred_2011.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/4_gt_2011.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/7_2011.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/7_pred_2011.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/7_gt_2011.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/14_2012.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/14_pred_2012.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/14_gt_2012.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/10_2011.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/10_pred_2011.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/10_gt_2011.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/10_2017.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/10_pred_2017.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/10_gt_2017.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/12_2017.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/12_gt_2017.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/12_pred_2017.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/5_2018.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/5_pred_2018.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/5_gt_2018.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/8_2018.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/8_pred_2018.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/supp_DIE_qualitative/8_gt_2018.png}
    \\ \noalign{\smallskip}

 \end{tabular}

 \caption{\textbf{Qualitative analysis of Text-DIAE in binarization task.} Images in columns are: Left: Original image, Middle: Binarized image using our proposed method., Right: Ground Truth image}
 \label{fig:our_results_dibcos_details}
\end{figure*}

\begin{figure*}[t!]
  \begin{center}
  \footnotesize
  \renewcommand{\arraystretch}{0.5}
  \begin{tabular}{cc}

    \includegraphics[width=0.4 \linewidth, height=40mm]{images/comparison_DIE_2017/3_deg_17.png} &
    \includegraphics[width=0.4 \linewidth, height=40mm]{images/comparison_DIE_2017/sauvola.png}\\\noalign{\smallskip}
    Original  & \cite{sauvola2000adaptive}\\\noalign{\smallskip}
    \includegraphics[width=0.4 \linewidth, height=40mm]{images/comparison_DIE_2017/kang.png}&
    \includegraphics[width=0.4 \linewidth, height=40mm]{images/comparison_DIE_2017/docentr.png}
    \\\noalign{\smallskip}
    \cite{kang2021complex}& \cite{souibgui2022docentr} \\\noalign{\smallskip}

    \includegraphics[width=0.4 \linewidth, height=40mm]{images/comparison_DIE_2017/3_pred.png}&
    \includegraphics[width=0.4 \linewidth, height=40mm]{images/comparison_DIE_2017/3_gt_17.png}

    \\\noalign{\smallskip}

     \textbf{Ours} & Ground Truth
    \\%\noalign{\smallskip}

    \end{tabular}
  \end{center}
  \caption{\textbf{Qualitative results of binarized samples:} We show the results of Text-DIAE on the document image binarization task. Given a degraded input example from DIBCO 2017, our model performs significantly better qualitatively compared to previous approaches.}
  \label{fig:binarization_qualitative_2}
\end{figure*}

\begin{figure*}[t]
\centering

 \begin{tabular}{ccc}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4005_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4005_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4005_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4010_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4010_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4010_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4015_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4015_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4015_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4024_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4024_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4024_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4192_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4192_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4192_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4667_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4667_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4667_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4830_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4830_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4830_gt.png}
    \\ \noalign{\smallskip}

    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4842_deg.png} &
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4842_enh.png}&
    \includegraphics[width=0.30 \linewidth, height=20mm]{images/deblur_qualitative_results/resized-image_4842_gt.png}
    \\ \noalign{\smallskip}

 \end{tabular}

 \caption{\textbf{Qualitative analysis of Text-DIAE in deblurring task.} Images in columns are: Left: Original image, Middle: Deblurred image output from Text-DIAE, Right: Ground Truth image}
 \label{fig:our_results_deblur_details}
\end{figure*}

\section{Introduction}\label{s:intro}
\input{./tex/intro.tex}

\section{Related Work}\label{s:sota}
\input{./tex/sota.tex}

\section{Method}\label{s:method}
\input{./tex/method.tex}

\section{Experiments}\label{s:experiment}
\input{./tex/experiments.tex}

\section{Conclusion}\label{s:conclusion}
\input{./tex/conclusion.tex}

\section*{Acknowledgments}
\begin{small}
This work has been partially supported by  the Swedish Research Council (grant 2018-06074, DECRYPT), the Spanish projects RTI2018-095645-B-C21, CERCA Program / Generalitat de Catalunya, the FCT-19-15244,  the Catalan projects 2017-SGR-1783, PhD Scholarships from AGAUR (2021FIB-10010) and (2019-FIB01233), and from UAB (B18P0073). DocPRESERV project (Swedish STINT grant).
\end{small}

\subsection{Text Recognition}
\noindent
\textbf{Evaluating Representations.} In order to evaluate the quality of the learned representations, and extending commonly used linear-probing settings~\cite{zhang2016colorful}, we employ a similar approach as introduced by~\cite{aberdam2021sequence}. As a first step, the encoder is pre-trained with unlabeled data as described in Section \ref{seq:pretraining}. After that, the encoder's weights are frozen and a new decoder is trained on top of it with all the labeled data. The decoder, as we detailed above, generates the predicted characters in a time-step manner. Since the encoder remains frozen, this scenario is a good proxy that represents the expressivity of the learned visual representations. To this end, Table~\ref{table:results_frozen_encoder} shows the results of our proposed approach. We compare among self-supervised methods specifically designed for the text recognition task.

\input{tables/two_tables}
\noindent
\textbf{Better performance.}
As it can be seen from Table~\ref{table:results_frozen_encoder}, the seqCLR method presented by~\cite{aberdam2021sequence} improves significantly a self-supervised baseline inspired by SimCLR~\cite{chen2020simple}. In the recently released approach PerSec by~\cite{liu2022perceiving}, they slightly improve over the seqCLR.
It is evident that our Text-DIAE model \textit{greatly} outperforms all the aforementioned state-of-the-art approaches regarding the representation quality obtained, both in handwritten and scene-text. The improvements in term of the accuracy in a handwritten text dataset, IAM, is close to \textbf{+20 points}. Moreover, a bigger improvement gap is obtained when evaluating scene-text. An average gain of \textbf{+30 points} is accomplished in IIIT5K and ICDAR13, proving the generalization of our method to different domains.
In our model, the great expressivity of features achieved by the encoder is mainly due to two factors.
Firstly, by masking image patches, the encoder learns a strong unigram character distribution (refer to Figure~\ref{fig:qualitative_ocr}), which is not leveraged in previous methods. Secondly, by  distorting and recovering  the image, we make the model learn richer representations to detect and recover the text into a clean and readable state. Thus, the model is learning the most valuable features that lead to the best recognition performance.

\noindent
\textbf{Faster convergence.} One of the most important outcomes by employing our method, is that a \textbf{paramount} improvement in convergence is achieved during pre-training.  Table~\ref{table:results_frozen_encoder} shows this effect under the column labeled as ``Seen''.  It  depicts the total number of seen samples that each model requires during the pre-training stage. It is worth highlighting that during pre-training the encoder of our model requires \textbf{~43} and \textbf{~166} times lesser data in IAM and CVL respectively when compared to the seqCLR and simCLR. In scene-text, our model employs only $18.2$M samples to yield powerful representations compared to the $409$M samples required by previous self-supervised approaches.

\noindent
\textbf{Fine-Tuning.} In this stage, we evaluate our model considering a semi-supervised setting where the obtained results are depicted in Table~\ref{table:results_fine_tuning}. Here we use the self-supervised pre-trained encoder as a backbone  and train a transformer-based decoder from scratch that predicts the characters in a sequential manner, as illustrated in Fig.~\ref{fig:method-ocr}-Left. In this scenario, the gradients are back-propagated not only to the decoder but also to the encoder. Following the previous work~\cite{aberdam2021sequence}, we use 5\% and 10\% of the labeled dataset by randomly selecting the training samples. As suggested in~\cite{chen2020simple} we perform fine-tuning on all the labeled dataset. In order to compare with~\cite{aberdam2021sequence} and since scene-text dataset is synthetic, we evaluate with the complete labeled dataset.

\input{tables/ablation_htr_objective}
\noindent
\textbf{Higher performance in fine-tuning settings.} Our model exploits data in a more efficient manner than previous self-supervised methods in fine-tuning setting. We infer that the set of degradations proposed yields rich signals, helping the encoder to adapt to the downstream task more efficiently. Our model achieves state-of-the-art in all scenarios when all the labeled datasets are used except in IAM where the PerSec is slightly better.
Under semi-supervised settings, our model performs better at the IAM dataset when employing 5\% and 10\% of the labels than simCLR and seqCLR. Since CVL contains substantially fewer data samples than IAM, SeqCLR still outperforms our approach in the CVL dataset. However, while employing the full labels of CVL, Text-DIAE outperforms all the methods by a large margin.

\noindent
\textbf{More efficient than a supervised baseline.}
From table~\ref{table:results_fine_tuning}, we can also notice the superiority of pre-training our architecture  compared to a fully supervised model starting from scratch.
This suggest that the self-supervised pre-training of such transformer-based architectures is essential to obtain better results, and beneficial  especially in small labeled datasets scenarios, since the unlabeled data is generally easier to obtain for a self-supervised pre-training.

\noindent
\textbf{The effect of fine-tuning after pre-training.} By proposing the degradation invariant optimization at pre-training, our model achieves a significant gain in recognition on handwritten text datasets. An average of 10 points of accuracy are gained after fine-tuning (refer to Table~\ref{table:results_frozen_encoder} and~\ref{table:results_fine_tuning}).
Finally, it is important to note that our model reaches state-of-the-art in the handwritten text recognition task, even comparing to specifically designed supervised approaches. The results on the IAM dataset are shown in Table~\ref{table:results_fine_tuning_iam}, which measures the performance of a model in terms of word and character error rate, WER and CER respectively.

\begin{figure}[t!]
\centering
\includegraphics[width=\linewidth]{images/recovering.pdf}
\caption{\textbf{Qualitative results of pre-training samples.} The left refers to handwritten text, while scene-text is depicted on the right. On each scenario, from left to right, the original, masked and reconstructed images are depicted.}
\label{fig:qualitative_ocr}
\end{figure}

\noindent
\textbf{Ablation Studies.}
The results of experimentation regarding the effect of each degradation as pretext task at pre-training is given in Table~\ref{tab:ablation_htr_objectives}. Firstly, among the three proposed degradations, masking is the most crucial to be applied in both tasks, handwritten and scene text recognition.
When an input word is masked, and in order to properly reconstruct it, the model has to learn a character level distribution.
This by itself provides with a strong prior compared to denoising or deblurring an image. Additionally, adding blur in scene-text imagery improves the representations learned by the model shown by the results. Lastly, adding noise does not result in an improvement in text recognition tasks. However, as it is shown in the next section, the combination of the $3$ degradation produce a richer encoder in document enhancement. Therefore, we can safely assume that each degradation has a task-dependent impact on the representations learned depending on the similarity of them when compared to the final downstream task and input data distribution.

\input{tables/two_tables_enhance}
\noindent
\textbf{Qualitative Results.}
In Figure~\ref{fig:qualitative_ocr} we show the reconstructed images at pre-training stage for handwritten and scene-text samples. It is important to note the complexity of the reconstruction task even for humans. Even though high masking percentages are employed (75\%), our model learns to properly adapt to handwritten styles and fonts found in scene-text. As can be appreciated, although sometimes our model's reconstruction does not match with the ground truth images, it can still reconstruct the most probable and plausible English words (e.g. see ``school'' vs ``sand'' in 4th row in handwritten examples). Another interesting outcome is also noticed for scene-text example where ``xperia'' is reconstructed correctly while the last character ``a'' is selected from another font, demonstrating the model's capability.
Minor reconstruction errors are found such as that the model eventually learns to overcome at fine-tuning stage.

\input{tables/dibco_table}
\input{tables/ablation_enhance_objectives}
\begin{figure*}[t]
\centering
\setlength{\tabcolsep}{0.31em}
\begin{tabular}{c c c c}
     \toprule
     \scriptsize{Original Input} & \scriptsize{DocEnTr~\cite{souibgui2022docentr}} & \scriptsize{\textbf{Ours}} & \scriptsize{Ground Truth} \\
     \midrule
     \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_orig.png} &  \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_docentr.png} & \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_diae.png} &
     \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_gt.png} \\
     \midrule
     \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_ocr_orig.pdf} &  \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_ocr_docentr.pdf} & \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_ocr_diae.pdf} &
     \includegraphics[width=0.23\linewidth]{images/qualitative_samples_ocr/Sample_tesseract_ocr_gt.pdf} \\
     \midrule
     \scriptsize{CER: 78.86} &  \scriptsize{CER: 18.51} & \scriptsize{CER: \textbf{8.94}} & \scriptsize{CER: 4.88} \\
     \bottomrule
     \end{tabular}
\caption{\textbf{Qualitative results of deblurred samples.} The document image on the left refers to the originally captured blurred image, followed by the ground-truth, and the deblurred results from the DocEnTr and our Text-DIAE model towards right. The correctly predicted OCR  output is shown in "Green" font while the inaccurate ones are depicted in "Red" and recognition performance in terms of CER.}
\label{fig:ocr_deblur}
\end{figure*}

\noindent

\subsection{Document Image Enhancement}

\noindent
\textbf{Performance Analysis on Binarization.}
As shown in Table~\ref{tab:dibco_sota}, the Text-DIAE outperforms the previous state-of-the-art approaches on majority of the standard metrics for document binarization task. Specifically, the quantitative comparison of results demonstrate that Text-DIAE achieves an optimal gain in PSNR, FM, $F_{ps}$ and DRD performance surpassing the all previous arts. The largest performance improvement is obtained over the H-DIBCO 2012 while the least performance gain is obtained in the H-DIBCO 2018.
One of the major concerns which degraded historical documents face is the show-through effect, which appears when ink impressions from one side of the document start appearing on the other side, making it almost impossible to read as shown in Appendix.
The enhanced Text-DIAE output
illustrates that it not only resolves the show-through but also sharpens and smoothens the edges of the foreground text approximately to the ground-truth level.

\noindent
\textbf{Performance Analysis on Deblurring.}
In Table~\ref{tab:results_deblurring} we show a quantitative comparison and superiority of Text-DIAE over supervised techniques ~\cite{hradivs2015convolutional,wang2018high, souibgui2020gan, souibgui2022docentr} on the document deblurring benchmark. A substantial gain in PSNR by \textbf{+2 points} on a \textbf{logarithmic} scale is obtained over DocEnTr~\cite{souibgui2022docentr}, which signifies the greater quality of deblurred images generated by Text-DIAE. There are two different kinds of blurring which appear in documents: motion blur owing to the sudden rapid camera movement and out-of-focus blur which emerges when light fails to converge in the image. In Fig.~\ref{fig:ocr_deblur}, we show an interesting qualitative case study of a motion blurred document image. We assess the performance of deblurring by running the Tesseract-OCR engine~\cite{smith2007overview} over the blurred, ground-truth, DocEnTr prediction and the Text-DIAE output. Qualitative results show that Text-DIAE significantly decreases the CER, showing vast improvement in OCR performance as depicted in green font.

\noindent
\textbf{Ablation Studies.} We also showcase an interesting ablation on the task of document image binarization for the challenging DIBCO 2018 benchmark.
From Table~\ref{tab:ablation_enhance_objectives}, we infer that any pre-training task is beneficial while the denoising task is the most crucial to be applied when each pre-text task is applied separately. The aforementioned result can be attributed to the fact that denoising is much closer to the downstream binarization task. Also, it demonstrates that Text-DIAE performs the best for document enhancement tasks when the model learns all the possible degradation (masking, blurring and adding noise) together.

\subsection{Pre-Training Module}\label{seq:pretraining}

The overall pre-training pipeline of Text-DIAE is shown in Fig.~\ref{fig:main_model}.
For each task, given an unlabeled image $I$ (eg. a cropped handwritten text, cropped scene text or a scanned document image
, we use a function $\phi$ to map $I$ to a degraded form. The function $\phi$ takes as parameters the original image $I$ and the degradation type $\mathcal{T} \in \{mask, blur, noise\}$ where
we denote a degraded image by $I_d = \phi(I, \mathcal{T})$.

Our model is composed of an encoder $\mathcal{E}$ and a decoder $\mathcal{D}$ with learnable parameters $\theta_\mathcal{E}$, $\theta_\mathcal{D}$ respectively. The pre-training pipeline trains an encoder function $\mathcal{E}$ that maps the degraded image $I_d$ to a latent representation $z_\mathcal{T}$ in a multi task fashion (unmasking, deblurring and denoising) and then learning a decoder $\mathcal{D}$ to reconstruct the original image $I$ from the representation $z_\mathcal{T}$:
\begin{equation}
\begin{split}
    z_\mathcal{T} &= \mathcal{E}(\phi(I, \mathcal{T}); \theta_E) \\
    I_r &= \mathcal{D}(z_\mathcal{T}; \theta_D)
\end{split}
\label{eq:pretraining}
\end{equation}
The learned visual representations from the latent subspace should be invariant to the applied degradation $\mathcal{T}$.
\input{tables/frozen_encoder_text}

\noindent
\textbf{Encoder.}
The encoder architecture consists of a vanilla ViT~\cite{dosovitskiy2020image} backbone. Given an input image $I_d$, it is first split into a set of $N$ patches, $I_d^p = \{I_d^{p_1}, I_d^{p_2},\\ ... , I_d^{p_N}\}$. Then, these patches are embedded with a trainable linear projection layer $E$.
Text-DIAE uses a distinct linear projection layer for every defined pre-text task. These tokens are later concatenated with their 2-D positional information embedded with $E_{pos}$ and fed to $L$ transformer blocks to map these tokens to the encoded latent representation $z_l$. These blocks are composed of $L$ layers of Multi-head Self-Attention (MSA) and a feedforward Multi-Layered Perceptron (MLP) as depicted in Figure~\ref{fig:main_model}. Each of these blocks are preceded by a LayerNorm (LN)~\cite{ba2016layer} and followed by a residual connection:
\begin{equation}
\begin{split}
    z_0 &=  E(I_d^p) + E_{pos} \\
    {z^\prime}_l &= \text{MSA}(\text{LN}(z_{l-1})) + z_{l-1} \text{, \space} l = 1, \text{ \dots L} \\
    {z}_l &= \text{MLP}(\text{LN}({z^\prime}_l)) + {z^\prime}_l \text{, \space} l = 1, \text{ \dots L} \\
    z_\mathcal{T} &= LN(z_L)

\end{split}
\label{eq:encoding}
\end{equation}

\noindent
\textbf{Decoder.} The decoder composed of transformer blocks following the same structure and number of layers as the encoder. The decoder input is
the output of encoder $z_\mathcal{T}$.
The output of the decoder is a set of vectors $I_{r} = \{I_{r}^{p_1}, I_{r}^{p_2}, ... , I_{r}^{p_N}\}$ where each of which corresponds to a flattened patch in the predicted (reconstructed) image. Same as before, a distinct linear layer is used for each pre-text task.
\begin{equation}
\begin{split}
    {z^\prime}_l &= \text{MSA}(\text{LN}(z_{l-1})) + z_{l-1} \text{ , \space} l = 1, \text{ \dots L}
\\
    {z}_l &= \text{MLP}(\text{LN}({z^\prime}_l)) + {z^\prime}_l \text{ , \space} l = 1, \text{ \dots L}
\\
  I_{r} &= \text{Linear}(z_L)
\end{split}
\label{eq:decoding}
\end{equation}

\subsection{Fine-Tuning}

Our fine tuning process is illustrated in Fig.~\ref{fig:method-ocr} where we perform two different downstream tasks; text recognition and document image enhancement.
\input{tables/finetuning_text}

\noindent
\textbf{Text Recognition.}
Text recognition aims to transform an image into the machine encoded form, i.e.,  sequence of characters.
Let $I$ be a cropped text image and $C_{}=\{c_{{}_1},c_{{}_2}, ..., c_{{}_N}\}$ its ground truth label which corresponds to a sequence of characters, where $N$ is the length of the text. The training is done by passing $I$ to an encoder function $\mathcal{E}$ to produce a latent representation $z$. Then, $z$ is later fed to a decoder function $\mathcal{D^{\prime}}$ to produce a sequence of characters $C_{p}=\{c_{p_1},c_{p_2}, ..., c_{p_N}\}$ that should match the ground truth label sequence.

We initialize the encoder with the pre-trained weights $\theta_\mathcal{E}$ while we employ a sequential transformer decoder~\cite{vaswani2017attention} as seen in Fig.~\ref{fig:method-ocr}-Left. The decoder is initialized randomly and composed of $L$ transformer blocks of MSA, MLP and Masked-MSA layers preceded by LN layers, and followed by a residual connection.
The output of the decoder is a sequence of characters where at each time step $t$, the predicted character is formed by attending to the representation $z$ and previous character embeddings until $t-1$.

\noindent
\textbf{Document Image Enhancement. } Document enhancement consists of mapping a degraded document into a clean form.
Let $I_d$ be a degraded image and $I_c$ its clean version, then the goal is to learn an encoder function $\mathcal{E}$ that maps $I_d$ to a representation $z$ with the same way as in Eqn~\ref{eq:encoding}. $\mathcal{E}$ weights are initialized from the pre-training stage. The decoder $\mathcal{D}^{\prime\prime}$ generates the clean image $I_c$ from $z$ as in Eqn~\ref{eq:decoding}.

\subsection{Learning Objectives}

Our model makes use of different sets of losses for each phase. During pre-training, we use three different losses. Each one is dedicated to a particular pre-text task: $\mathcal{L}_{mask}$, $\mathcal{L}_{blur}$ and $\mathcal{L}_{noise}$. Each of these losses is a mean squared error (MSE) between the reconstructed image $\mathrm{I}_{r}$ (from the masked, blurred or noisy image) and its ground-truth version $\mathrm{I}_{gt}$.
Thus, the overall loss for our pre-training stage is:
\begin{equation}

    \resizebox{0.45\textwidth}{!}{$\mathcal{L}_{pt} =\lambda_{1}\mathcal{L}_{m}\left(\mathrm{I}_{r}, \mathrm{I}_{gt}\right)+\lambda_{2} \mathcal{L}_{b}\left(\mathrm{I}_{r}, \mathrm{I}_{gt}\right)
    +\lambda_{3} \mathcal{L}_{n}\left(\mathrm{I}_{r}, \mathrm{I}_{gt}\right)$}
\end{equation}
During our experimentation, the best results were obtained with setting $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$.

While fine-tuning on text recognition, we use a cross-entropy loss between the predicted sequence of characters $C_p$ and $C$.
For document image enhancement fine tuning, we used an MSE loss between the cleaned image $I_c$ and $I$.

redpajama_set_name: RedPajamaArXiv


text:
Heroes Inven
Heroes/Talent Calculator
Author : Tim Rizzo
Biggest takeaways from the Heroes of the Storm AMA
Tim Rizzo
On Thursday afternoon, members of the Heroes of the Storm development team sat down on the game's official subReddit (r/HeroesoftheStorm) to conversate with members of the community. They answered fan questions, provided insight as to what's the come and continued their mission to remain as available to the public as possible. With hundreds upon hundreds of questions asked, it can be difficult to keep track of everything that was said. Luckily, InvenGlobal has highlighted a few of the moment noteworthy responses below to save readers from having to scroll through hundreds of replies for the juicy ones.
Developer insight into the delicate balance behind Chen's rework
You'd be hard-pressed to find a character in Heroes of the Storm over the last few months that have had more requests to be reworked than Chen. The only "Pandamonium" that he has brought to the battlefield for the last couple years has been to the player piloting the panda who has a fondness for drinking brew. More often than not, Chen didn't feel impactful when dropped on the map. His death-by-papercuts damage output was a minor nuisance; his over-reliance on consuming brew to stay alive in fights wasn't particularly interactive; and when it came to the solo-lane, he was out-classed by many.
How the Heroes' community became heroes for one player
It's been nearly two weeks since it happened. Nick Young, a 22-year old from Virginia, hasn't touched Heroes of the Storm since. He has too much going on inside his mind. Funeral arrangements, a new living situation, everything around him rehashing old memories that cause him happiness and push him to tears with agony. They say the grieving process varies based on the individual. However, in recent days, Young has found comfort, support, and love within the unlikeliest of places. A location he can see but not touch. Feel the arms of others holding him but not hug back.
Bakery aims to shatter the mold on what a career in esports looks like
Currently, the Business Development Manager of Dignitas, one of the largest esports organizations in the world, Bakery's day to day roles drastically differ from when he rose to the top of the Heroes of the Storm scene during his playing days. Instead of grinding matches, dissecting replays and mastering his mechanical craft on the keyboard, Bakery's life now consists of overseeing ways to monetize the company via their website, merchandise, and social media platforms. Upon making an abrupt retirement announcement at the end of 2017, Bakery set his sights bigger than just being good at playing a specific game better than almost anyone else in the world.
Rutgers crowned Heroes of the Storm Tespa Collegiate Champions
There's a new champion in the world of collegiate Heroes of the Storm esports. In a highly-contested series, Rutgers University took down Cal Poly Pomona 3-1 in the grand finals of the Tespa Collegiate Series on Sunday afternoon at the George R. Brown Convention Center in Houston, Texas. The roster consisted of Dominick "BegForMercy" Maisano, Rick "hiimrick" Green, Hexuan "NatsumeAl" Ma, Dustin "Supertai" Nguyen, Peter "noOreos4u" Thach and Stan "msMarvel" Potapov. Each player, in addition to a championship trophy and bragging rights, will receive a $500 scholarship prize to their respective university and one ASUS ROG laptop.
Four biggest takeaways from the Heroes of the Storm AMA
On Thursday afternoon, members of the Heroes of the Storm design team took to the game's official subReddit (/r/heroesofthestorm) to answer questions fans had and talk about the future of the title itself. For those who want to peruse the full thread, the AMA can be found here. If you're looking for the most, in my opinion, important bits of information scattered throughout the over 1,200 comments on the AMA, the four most noteworthy topics are laid out below that shed some light on the future of the title in 2019.
A week in review: Heroes is a Classic, Cloaken leaves, a dank panda.
Fans of the Heroes of the Storm franchise went through a lot this past week. From weird highs to depressing lows, the full spectrum of emotions was experienced. Maybe it's fitting that the end of this week fell on 4/20 as the "holiday" can allow individuals who celebrate to cope with the pain or comprehend the oddness that occurred. An icon left the company, Heroes of the Storm is now labeled as a "Classic Game" and more took place that left us thinking...what?
Heroes Devs on Chromie: "Hopefully, with these changes, we don't have a plan to rework Chromie again in six months..."
Chromie and reworks: name a more iconic duo. For the third time in the past 18 months, the long-ranged Assassin in Heroes of the Storm has been reworked to provide an overall better playing experience for those piloting and playing against her. The newest iteration of her kit, which can be found here, is one that the design team hopes might stand the test of time. Most Heroes in the game have not experienced drastic reworks, yet Chromie continues to receive attention from the development team. I spoke with Alex Neyman, Senior Live Designer, to shed some light on why it's, once again, Chromie's time in the re-design spotlight.
The holistic Hero rework: Brett Crawford on Heroic parity, a Hero's feel, and more.
Last week the Heroes of the Storm development team revealed the two newest heroes to receive an overhaul or "rework" to their existing in-game kits. One of them, the Support hero, Lucio, was released into the game back in February of 2017. Upon his entrance into the nexus, his healing output--thus his power level--was quite high. All a player had to do while manning the character was stay within range of his teammates while his healing and speed auras would do the rest of the work. Occasionally, if the team was in trouble, his Sound Barrier heroic would provide a massive burst heal that would either turn the tide in a fight or save the team long enough for them to scurry away to safety. Over time, through tweaks to his kit, the design team provided a more interactive experience when playing with and against Lucio. But, it wasn't enough.
Khaldor on META Madness: "The draft alone should be great to watch from a players perspective."
Beginning this weekend, the META Madness tournament hosted by long-time Heroes of the Storm caster Thomas "Khaldor" Kilian will kick off with six teams fighting it out for a €1,000 prize pool. The event is being sponsored by RUNTIME, a nutrition company based out of Europe. Taking place over a two-day span, the bracket-style tournament will feature best-of-five matches until a winner is crowned. What separates this event from the other Heroes' tournaments that have popped up since Blizzard stated they would no longer support the competitive Heroes of the Storm scene is the "Madness" aspect. According to the rules, heroes that are banned during the Draft phase are not eligible to be selected for the entire tournament. With numerous top-tier threats off the board, players will need to adapt over time and test their abilities to perform with unique and off-meta heroes.
18 months, 3 reworks, 1 Chromie: a patch notes story
Ever since Chrome, the Keeper of Time, went live in Heroes of the Storm back in May of 2016 the development team has had their hands full finding out what they want to do with her. A Ranged Assassin who can one-shot an enemy with a couple of well-timed abilities is a delicate balancing act. In theory, if a hero relies on landing long-range skill shots for the majority of their effectiveness in battle the balance team needs to consider a few design aspects: It's important to make the damage output impactful enough to make up for that character's low health pool while rewarding a good player who is able to land difficult skill shots. If the range on those abilities is too long and the damage is too high, they will be able to poke away at the opposition while remaining at a safe enough distance, providing very little counter-play and a frustrating experience for the opponent. If the opposite is true then players will not feel rewarded for playing that hero if they can be engaged upon very easily, blown up and lack the necessary tools to keep themselves safe while not putting out enough damage to make the risk worth the reward. In the past 18 months, Chromie has experienced three reworks to her kit, including two in the past seven. By following the clues left in the patch notes since her inception and developer comments left by the balance team, a tale can be told about what they may have intended her to be and what they learned along the way.
Lucio full reworked talents and abilities
One of the grooviest heroes in the Nexus, Lucio, is the newest recipient of a rework in the latest Heroes of the Storm Public Test Realm as the patch went live Monday morning. Since he was introduced into the game in February of 2017, Lucio's popularity--similar to his rhythmic jams--have had its' ups and downs. There was a time when he was arguably the strongest Support character and helped push the Double Support meta into existence. Then, once the development team knocked Support characters down a peg as a whole at the end of 2017, his relevancy disappeared along with it. Times are changing, however. The development team ripped apart his outdated tools and reassembled it in a way that makes him more interactive to play with and against.
The most violent team wipe in recent memory broken down
If you blinked, you missed it. In a six-second span, an elite display of professional game knowledge, team coordination and mechanical skill was put on display during the Heroes Lounge Nut Cup over the weekend that left Twitch chat, Reddit and the Heroes of the Storm community with their jaws on the floor. In Game Two of their series, team "Washed Up" consisting of former professional players Thomas "Ménè" Cailleux, Dennis "HasuObs" Schneider, Filip "SmX" Liljeström, Aleksandar "ethernal" Milanov and Ivan "SportBilly" Koturić pulled off a team wipe of "Coming Through"'s roster that all but sealed the match and series.
Despite cutting costs, HotS dev team is as dedicated as ever
They didn't make the decision to have their employer, Activision Blizzard, cut support to the competitive scene. They certainly didn't have a say in numerous individuals who were working on the game being transferred to other titles, making their lives more difficult. And, they didn't intentionally alienate thousands of players who have slowly stopped playing the game over the last four months. What is in their control, is how the inner workings of the game itself will operate going forward to ensure that the current playerbase is given the best playing experience possible. On Wednesday afternoon, members of the development team held an "Ask Me Anything" (AMA) on the game's subReddit (/r/heroesofthestorm) where fans were chomping at the bit to get the opportunity to have their questions answered.
Collegiate Heroes of the Storm tournament reaches five-digit viewership
What transpired over the weekend was a spectacle within the Heroes of the Storm community that many feared would never happen again after Activision Blizzard stated they would no longer support the game's competitive scene back in December. A competitive Heroes of the Storm tournament reached over 11,500 --11,537 to be precise-- viewers on Twitch at a single moment, according to twitchtracker.com.
Four months later, first Korean HotS tournament since BlizzCon announced
Ever since Blizzard stated they would no longer be supporting a competitive Heroes of the Storm professional circuit, numerous crowd-funded tournaments have been popping up within North America and Europe. Now, Korea is getting involved in the action. On Thursday long-time Korean commentator Daniel "Gclef" Na shared on the Heroes of the Storm subReddit that a tournament featuring former pros was in the works. Today, specifics have been revealed. The event titled, "Heroes of the Storm League: REVIVAL" will begin on March 14, featuring a $6000 prize pool and includes eight teams made up of some of the most well-known and accomplished players to ever play the game.
How Blizzard can monetize Heroes of the Storm going forward
Activision Blizzard loves making money. So much so, that their Chief Executive Officer, Bobby Kotick, during their most recent earnings call began his statement with: "We once again achieved record results in 2018." What followed was him stating that the company would be laying off around 800 individuals but that's a different story altogether. A few of the company's entities: Overwatch, the Overwatch League, World of Warcraft and Hearthstone have reliably brought in income for the company in recent years. What has struggled-- and likely one of the reasons why developers were moved off the title and onto more profitable ones-- is a way for Heroes of the Storm players to be incentivized to spend money in or on a title they love. This isn't a recent development either. All a die-hard fan of the game has to do is visit Blizzard's official gear shop online or at a company-sponsored tournament and you will see very, very little Heroes of the Storm merchandise. Why this is so perplexing is the concept of Heroes lends itself well to merchandising.
4 Games to play during oppressively long Heroes of the Storm queue times.
We've all been there. You download the newest Heroes of the Storm patch on the launcher, anticipating the opportunity to jump into a Ranked game and try out the newest tweaks on your favorite heroes. You rush to click on "Hero League" or "Team League" when, like a Death Eater from Harry Potter, an estimated queue time of 803 seconds sucks the excitement out of your body. What's there to do for over 13 minutes? You could pray to the Lord of Sin, Azmodan, and hope the stars align in a way that the wait is only a couple of moments. Or, if you're truly cursed, the 803 seconds is a conservative estimation and, depending on your rank, you may be sitting in the queue for upwards of 20 minutes. You could cancel your attempt to play in Ranked mode and wait only a few minutes to play Quick Match or Unranked but, eh, that's not your cup of tea. You're a competitive fiend whose skin tingles when you see +3 for Performance Adjustment at the end of a hard-earned win. The good thing is that, while I cannot make your queue time any faster, I can present a list of cheap or free games that you can play in queue opposed to having your mind slip into madness as you watch your two favorite heroes spin in a circle as you wait for a game to begin.
Heroes of the Storm's Twitch viewership plummeting
When Activision Blizzard announced that the company would no longer financially support the competitive Heroes of the Storm scene back in December, the community was split. There were those who didn't particularly care as following the competitive scene wasn't their cup of tea, others were concerned this would be the end of their beloved title and a wide range of feelings in between existed as well. Two months into a world without the Heroes Global Championship (HGC), a story is unfolding that may be an indication as to what the future could look like for the title on Twitch, the largest streaming platform in the world.
Who is Anduin Wrynn? Heroes of the Storm's next hero(?)
Late Tuesday night, a Korean member of the Blizzard Entertainment scene leaked some information via datamine related to the next hero slated to enter the Heroes of the Storm universe. Lord Anduin Llane Wrynn, the High King of the Alliance, King of Stormwind, son of Varian Wrynn, nephew of Jaina Proudmoore and nephew of Magni Bronzebeard, among the other numerous titles he posses, is rumored to be entering the nexus.
For Love or Money: The Motivation of the Modern Esports Player is Changing
UNIQLO UT Line of Blizzard T-Shirts Launches Today
Heroes Live!
FanHOTS 861
AliciaWins 118
Malganyr 99
Kiyeberries 88
KidKerrigan 75

redpajama_set_name: RedPajamaCommonCrawl


text:
High School of Dublin (Dublin, Munster, Republic of Ireland)
Dublin, Munster, Republic of Ireland
Ancient works performed here:
Oedipus the King
Aias (Ajax)
Philoktetes
Bacchai
Productions at this venue
Filter productions
Performed after
Format: 2022
Performed before
Ancient plays
Ancient works performed
Medea (1957) 1957 High School of Dublin Medea
Antigone (1958) 1958 High School of Dublin Antigone
The Bacchae (1959) 1959 High School of Dublin Bacchai
Cyclops (1960 - 1969) 1960 High School of Dublin Cyclops
Ajax (1960 - 1969) 1960 High School of Dublin Aias (Ajax)
Frogs (1960 - 1969) 1960 High School of Dublin Frogs
Philoctetes (1960 - 1969) 1960 High School of Dublin Philoktetes
King Oedipus (1960 - 1969) 1960 High School of Dublin Oedipus the King
How to cite this venue
High School of Dublin (Dublin, Munster, Republic of Ireland), accessed at http://www.apgrd.ox.ac.uk/productions/venues/2616 <29 January 2022>


Israeli soldiers abducted, on Saturday evening, a child and a young man from Qalandia refugee camp, north of occupied Jerusalem, and south of Ramallah, while Illegal Colonists attack cars, west of Ramallah.
Media sources said the soldiers stopped and interrogated the two Palestinians, who were walking near the Qalandia Terminal, and abducted them.
They added that the abducted Palestinians have been identified as Mohammad Awni Aslan, 16, and Abdul-Malek Hasan Abu Latifa, 19.
In related news, illegal Israeli colonists hurled stones at many Palestinian cars near Ras Karkar village, west of Ramallah.
The cars were driving on Wadi el-Delb road, near Ras Karkar village, when the colonists attacked them, causing damage.
After the attack, Israeli soldiers closed the road to Palestinian traffic, all the way to Ein Ayyoub Junction, west of Ras Karkar, while Israeli colonists were not even stopped and continued to use the road.
In related news, dozens of illegal colonialists hurled stones at Palestinian cars driving at the junction of Gush Etzion colony, north of Hebron, in the southern part of the occupied West Bank, and shouted insults and threats at the Palestinians.


Movember is over – thank you!
Movember (the month formerly known as November) is over. I spent 30 days cultivating a moustache to promote the cause of men's health. I was motivated by the case of my father who was diagnosed with potentially terminal prostate cancer earlier this year.
Thanks to many of you I raised £766 to fight prostate cancer. Among the sponsors, to my astonishment, two people I have never met (via Facebook and Twitter).
Some of you have asked about my father. He had a course of radiotherapy over the summer, and then a fresh PSA test a few weeks ago. The level was extremely low. With the caveat that PSA tests are notoriously unreliable, the result suggests his prostate cancer was virtually undetectable. We are heartened by this, of course, but we continue to pray for his health.
I recently passed a symbolic milestone: my 35th birthday. Psalm 90 suggests I'm halfway through the days of my years. The proverbial bus could kill me tomorrow of course; and perhaps "reason of strength" will prolong my years. But I can't deny the milestone causes a little reflection.
British life expectancy is 79.4, almost four score years. But most of the world's population will not live to 70. The global average life expectancy is just 67.2. In Swaziland, it's just 39.6. If I was born there, I might now be in the twilight of my years.
My father was diagnosed with an aggressive prostate cancer earlier this year. Without asking his consultant, he was told rather bluntly he had about four years to live. He is 65, 30 years but a day older than me. He was preparing to enjoy his twilight years: scaling back his work, sailing, playing music and enjoying more time with his family, including a young grandson.
But my father's fate feels imminent. It looms on the near horizon for him, my mother and the rest of those close to him. He doesn't want to leave us, and we don't want to lose him. Life is short. We must all face our own mortality; but it is a tough blow, at any age, to learn your remaining years are numbered on the fingers of just one hand.
Perhaps the prognosis is wrong. The celebrated case is that of the freed Lockerbie bomber, Abdelbaset Mohmed Ali al-Megrahi: still alive two years after a medical prognosis of three months. But he will die with terminal cancer and my father will too. Despite our prayers, it will almost certainly kill him.
Cancer of the prostate is the most common cancer in men. One in nine men will be diagnosed with this cancer at some point in their lives. In 2008, 37,051 new cases were diagnosed and 10,168 men died from it. Incidentally, lung cancer affects fewer men, but kills many more.
Most victims survive prostate cancer. Indeed it poses no significant threat to most of the men it affects. Others are treated successfully, but a quarter cannot be cured.
Prostate cancer is much more likely to strike older men, especially after the age of 45. Those whose brother or father are affected are twice as likely as others to develop the cancer themselves. Other risk factors include obesity, low levels of vitamin D, high alcohol intake, and elevated blood levels of testosterone.
Because of the familial risk factor, in a few years time, I must decide whether to be tested for potential prostate cancer.
Currently, the primary early test is the prostate specific antigen blood test. But the PSA test is controversial. The NHS insists doctors discuss the pros and cons with any patient prior to testing. The PSA test is unreliable and risky. It sometimes fails to reveal cancer, it sometimes suggests the presence of cancer when there is none and it sometimes reveals harmless cancers, which may lead to unnecessary and perhaps even counterproductive treatment.
Due to the risks, there is no prostate cancer screening programme in the UK. But there are obvious risks too, in not testing. Early treatment is normally more successful, and that is only possible when a cancer is known.
A new urine test for the engrailed-2 (EN2) protein may offer fresh hope. Possibly as easy as a pregnancy test, scientists at the University of Surrey believe it is twice as effective as the PSA test. Pending wider trials, and cautious voices, the EN2 method could be available in a year's time.
However, a biopsy is the only test which can fully confirm a prostate cancer diagnosis.
I want to promote the fight against prostate cancer and help raise funds to fight it. I will cultivate a moustache throughout the month of November to be a living embodiment of the 'Movember' cause.
I will shave my upper lip for the last time on the evening of 31st October; and then not again until 1st December. As an introvert, I am a little nervous about walking the streets of Geneva with a baby mo. But it will be a worthy endeavour… I promise!
Please visit my Movember page and give a little spare change.
Prostate cancer remains relatively low in public consciousness. I feel it needs the attention, and the issue certainly needs resources.


It was 16 years ago when I placed my only child for adoption, the hardest and smartest thing I ever did.
I was 34, and for 14 years, I didn't think I could get pregnant. So when I got the positive dollar store test, I was shocked, to say the least.
I went to Planned Parenthood to confirm the test several weeks later. A few weeks later, I moved into a maternity home in Fort Worth, Texas on January 31, 2000.
On February 28, I got an ultrasound, which showed I was having a daughter. Though I slipped into a deep period of depression while I was pregnant, I knew I had finally grown up because I was making the first unselfish decision of my life. I knew that as badly as I wanted to have a child, I could not parent her for emotional, financial, mental, and physical reasons that had plagued me all my life.
Shortly after I moved into the dorm at the maternity home, I started an adoption journal, and in my regular therapy sessions I would cry from grief about my decision, even though I knew it was the right thing.
Throughout my pregnancy, there were times I thought that if one more person asked me why I was placing my child for adoption, I'd choke them. I didn't know if the pregnancy was making me stronger, but I found myself standing up to people, which I'd never been able to do.
Two months before I had my birth daughter, McKenna, I met the people who would become her adoptive parents. One month later, I met her adoptive brother, who had also been adopted two years earlier.
Then on August 24, I placed my daughter with her new family on what is known as Placement Day. Pictures are taken, videos are made, gifts/cards are exchanged, and it is a total ceremonial setting.
I have a semi-open adoption, which means I get letters, videos, pictures, cards, gifts, emails, and I send the same. I made a scrapbook over a four-month period for McKenna before she was born about my life, and I write her letters on significant holidays and birthdays every year.
I also get to see her two or three times a year. It is as if a picture that I have in my bedroom of McKenna and I has been freeze-framed in my mind and soul and comes to life during our visits.
It used to be that the sound of a bunch of girls' laughter would echo as I left a store, and I would wonder if I'd ever get through a day when the sound or the sight of a girl didn't jerk at my numb heart or threaten to stir up tears. Now, when I hear a girl's laughter or voice that is the same age as McKenna, I smile inside and wonder what she's doing today.
The first holiday season, McKenna's first Thanksgiving and Christmas, was brutal for me. I cried in the bathroom on Christmas Day as my family prepared to eat while I was visiting my sister in Florida.
Then one day, I could finally see the light at the end of the tunnel a little and focus on how happy McKenna was.
If I had kept her, I would have been on welfare, but being bipolar with a lot of other emotional problems, my lack of financial resources would have been the least my problems.
Shortly after I had McKenna, I wished I could hold her for at least a minute or even a day, but I knew as soon as she got fussy I'd panic and look for her adoptive mom to take her.
My sister, who's a therapist, once told me that it is mature to admit that you know you're immature.
I got so much grief at work for my painful adoption decision that when I changed jobs, and when someone asked if I had custody of McKenna, I told them I did, only later admitting the truth.
Then on April 15, 2001, almost eight months after Placement Day, the adoptive mom asked me if I'd like to see McKenna on the day they were scheduled to go to court and have her adoption finalized. For the first time since Placement Day, I got to see and hold McKenna, and we took pictures in which I looked my happiest ever in my life.
I started talking to others about the adoption more, and I was not ashamed.
Fast forward to 2009, and in July, I was invited to my birth daughter's house by her adoptive mom. To be in McKenna's home where she lives with her family, to see how she lived and how she was, was a gift, and I cannot adequately put into words how much it meant to my heart, mind, and spirit.
I can't imagine my life without McKenna now.
Terri Rimmer has 33 years of journalism experience, having worked for ten newspapers and some magazines. She wrote for associatedcontent.com, later bought out by Yahoo Voices from 2005-2012. Ms. Rimmer published her e-book "MacKenzie's Hope" on booklocker.com under the family heading. It's also listed on adopting.com. On July 6, 2017, her story "The Birth Mom With No Regrets" was published by New York Magazine (The Cut) and in March 2017, her foster care story was published on "Foster Kids, Tell Me Your Story's" Facebook page.
Beautiful! We think about our daughter's birth parents all of the time and are so happy to have an open adoption. I can only hope they are feeling the way you feel. It's definitely a difficult, emotional journey. Hugs!!
Thank you so much! It was great to see someone commented so positively!


text:
Home > Arts & Entertainment > NAMI offers Native film showcase
NAMI offers Native film showcase
by From Press Release - November 15, 2021 November 15, 2021
"Run Woman Run" is among the films in NAMI's Native Cinema Showcase. (Courtesy photo)
The National Museum of the American Indian's Native Cinema Showcase, a celebration of Native film, continues through Nov. 18. The showcase focuses on Native people boldly asserting themselves through language, healing, building community and a continued relationship with the land throughout the Western Hemisphere and Arctic.
A series of panels with Native filmmakers and writers about all aspects of Indigenous storytelling from their own experiences is also available on demand.
The Song of the Butterflies (El Canto de las Mariposas)
Available on demand from 1:01 a.m. Nov. 15 to 11:59 p.m. Nov. 16,
(Peru/Colombia, 2020, 65 min.)
Director: Núria Frigola Torrent
Spanish and Munuka with English subtitles
Run Woman Run
Available on demand from 1:01 a.m. Nov. 17 to 11:59 p.m. Nov. 18
(Canada, 2021, 100 min.)
Director: Zoe Hopkins (Heiltsuk/Mohawk)
Inhabitants: An Indigenous Perspective
Available on demand from 12:01 a.m. Nov. 12 to 11:59 p.m. Nov. 18
(USA, 2020, 76 min.)
Directors: Costa Boutsikaris and Anna Palmer
Producer: Ben-Alex Dupris (Colville)
Rez Metal
(USA/Denmark, 2021, 76 min.)
Director: Ashkan Soltani Stone
Assistant Director: Veronica Quam (Zuni)
Associate Producer: Pamela J. Peters (Navajo)
From Press Release
Hard Rock sets its sights on Chicago casino
CSSS to host drive-thru 'pop-up' Dec. 1


When the owner of this 19th-century Normandy village house first moved in three years ago, he thought he would simply give the place a coat of paint. After all, it had been completely restored, including the construction of a 1,000-square-foot addition, in the 1980s. Everything seemed in relatively good shape. But upon closer inspection of the master bath, he saw that beneath the marble was pretty wood paneling. He called on interior designer Franz Potisek, who had also renovated the owner's Left Bank apartment in Paris, and together they decided to rip out the marble and tile except for the flooring, replace all the fixtures, and repaint the paneling in light colors, so it looked "like it was the house's original bathroom," Potisek says. The problem was, once that was done, "the rest of the house looked awful in comparison," he says with a laugh. "So we kept going."
In the drawing room of a 19th-century Normandy house decorated by Franz Potisek, the 19th-century armchairs are French, the cocktail table is custom made, and the curtains are of a plaid wool by Sanderson. The door and walls are painted in Zuber's Matte in Noir de Fumée, Femme Chocolate, and Frau Gretschen, and the beams are in Farrow & Ball's Estate Emulsion in Eating Room Red.
The key to the makeover was the use of color—a lot of it. The owner suggested "green with touches of violet, or blue with touches of red," Potisek recalls. "But I didn't want it to be rustic, with plain wood, like you see in most country houses. I find that sad."
Potisek designed the dining room cabinetry, the late-20th-century chairs are Spanish, the French chandelier is from the late 19th century, and the early-20th-century rug is from the Middle East.
The designer prides himself on being a colorist and wanted to make a stronger statement: "People often use beige and off-white because they are afraid to use color, but I am not. I think color is life!" He suggested they use all four shades—blue, red, green, and violet—but muted. And throw in some pink, too, "like faded flowers."
The armchair in the library is by Nils Holger Moormann, the sculpture above the armchair is Indonesian, and the light fixtures are by the Conran Shop.
In the living room, Potisek painted the woodwork two tones of celadon and the beams a dull red, and covered the armchairs in a smoky blue and a purple. The curtains are the same blue and green as a Scottish plaid—the owner's suggestion. They also used plaid fabric—in blue—for the upstairs hall wallcovering. "I love plaid because it gives the idea of comfort," says the homeowner. "For me it is very country."
The oversize fireplaces, which were new when installed in the 1980s, were a big draw for the owner. "They make the little rooms feel so warm," he says. But Potisek found them a bit too nouveau; he applied a patina finish so they appear to be the same age as the house.
In the master bedroom, the walls are covered in a Jasper hemp, and the chair rail and trim are painted in Zuber's Matte in Payne.
The kitchen was gutted and replaced—all that remains are the original terra-cotta floor tiles and the beams, which they left exposed and painted white. The cabinetry and fixtures are vaguely retro. "We designed it so it looked like it had been there forever," Potisek says.
Portraits bought on eBay in the corridor outside a guest room; the wool fabric on the walls is by Holland & Sherry, the armchair came from a flea market in Saint-Ouen, and the rug is by Crucial Trading.
The decor has that same timeless feeling. The owner bought much of the furnishings and artwork on eBay and at the Paris auction house Drouot. But he also found pieces, like the metal towel rack in the master bath, at flea markets near Avignon in Provence.
The tub in the master bath is by Jacob Delafon with fittings by THG, the floor tiles are by Winckelmans, and the walls are painted in Farrow & Ball's Estate Emulsion in Off-White.
The goal, says Potisek, "was to give historical integrity back to the house, to make it appear as if we didn't do any work so that nobody wonders what period the house was done. That's when we know we have done a good job."
ORIGINAL TEXT BY CATHERINE SCOTTO. Originally published in ELLE DECORATION France.
The pergola and Adirondack chair are painted in Zuber's Payne.


text:
Home / News & Media / Publication of the Final Report of the Royal Commission into the Management of Police Informants
Publication of the Final Report of the Royal Commission into the Management of Police Informants
The Australian Bar Association welcomes the publication of the Final Report of the Royal Commission into the Management of Police Informants. The implementation of the recommendations in the report will enhance public confidence in the administration of justice. The ABA will work with the Legal Services Council and the Law Council of Australia to implement the recommendations in the Final Report relating to it, particularly in relation to harmonising the duties of confidentiality and continuing professional development for barristers and solicitors across Australia."
Matthew Howard SC
News Areas


OSIR Design O-Shield is for Apple iPhone 3G & 3G S.
The O-Shield is made up of 4 layers of prepreg Carbon Fiber (Dry Carbon Fiber).
Very strong for everyday use. Simply snap over your iPhone 3G to install.
No needed double sided tape or adhesive. Ultra lightweight, weighs only 1/2 oz.
Fits iPhone 3G & 3G S only.
Free US fast Shipping 5-10 days.


Washington Dulles International (IAD) in Washington, DC.
Outside of the Main Terminal on the east end of the Departures level. Go out the doors closest to the United Airlines ticket counter and turn right.
Outside of the Main Terminal on the west end of the Departures level near the walkway to Daily Garage 2. Go out the doors closest to the Delta ticket counter and turn left.
Outside of the Main Terminal on the east end of the baggage claim level. Follow signs near Baggage Claim #1 which say "Service Animal Relief Area."
A and B-Gates: Between the concession area and Gate A-31.
C and D-Gates: Across from Gate D-1.