text:
\section{Introduction}

With the advent of deep learning, reinforcement learning (RL) with function approximators has become a promising strategy for learning complex behavior \citep{silver2017mastering, mnih2015human}. While RL can work remarkably well, deep RL experiments have been observed to exhibit high variance \citep{chan2019measuring, clary2019let, lynnerup2020survey}. Specifically, the rewards obtained by RL agents commonly vary significantly depending on the random seeds used for training, even if no other modifications are made. Implementation level details often have an outsized effect on performance \citep{engstrom2020implementation} and even hardware scheduling can significantly affect outcomes \citep{henderson2017deep}. In comparison to RL, computer vision and natural language processing (NLP) experiments often have lower variance \citep{bouthillier2021accounting}.

High variance is problematic for creating reproducible RL research, and achieving statistically significant results over large RL benchmarks is often prohibitively time-consuming  \citep{colas2018many, agarwal2021deep}. Furthermore, this variance provides an obstacle for real-world applications of RL, where safety and predictability can be critical~\citep{garcia2015comprehensive, srinivasan2020learning}. Relevant applications where RL is already being evaluated include medical testing \citep{bastani2021efficient}, autonomous vehicles \citep{bellemare2020autonomous} and communications infrastructure \citep{liu2020deep}. RL agents are typically evaluated on the average performance, but for these sensitive real-world applications, the drastic performance drop for the worst-performing training runs could yield disastrous outcomes.

In this paper, we aim to investigate causes for high variance in RL.
To limit the paper's scope and computational footprint we focus on continuous control from pixels with a SOTA actor-critic algorithm \citep{lillicrap2015continuous, yarats2021mastering}. This is a popular setup that exhibits high variance, and where actor-critic algorithms often give SOTA results \citep{laskin2020curl, laskin2020reinforcement, kostrikov2020image}. Additionally, this class of tasks is relatively close to real-world robotics \citep{zhan2020framework}.

In this setting, we first show that variance often is dominated by outlier runs that fail early in the learning process and never recover. It is natural to hypothesize that the typical sources of randomness,
network initialization and initial exploration, are to blame.
However, we demonstrate that these do in fact have little impact on the final performance.
We similarly provide experiments that suggest that neither poor feature representations nor excessive sensitivity of the environment are at fault. We instead demonstrate that numerical instability can drive most of the variance and show that exploding activations that saturate action distributions can often yield poor performance. We propose a few methods to avoid these issues and find that they do indeed decrease variance and improve average performance. The most effective method is simply normalizing the penultimate features which avoids exploding activations by design. We further show that our proposed approaches reliably enable larger learning rates and decrease variance significantly across tasks.
In fact, by combining multiple approaches we can reduce the variance by close to two orders of magnitude on a humanoid locomotion task.
We conclude that simply addressing numerical stability in the network parametrization is a promising avenue for stabilizing RL agents. Our experiments also suggest that high variance is not necessarily inherent to the RL setting. We summarize our contributions:
\begin{itemize}
    \item We provide extensive experiments on the variance of a popular continuous control setup. We show that outliers often dominate the variance, but that poor initialization, sensitive environments, and feature learning are likely not the issues.
    \item We show that saturating action distributions and exploding activations can cause outlier runs and propose several methods to avoid such cases.
    \item We demonstrate that the proposed methods significantly decreases variance while improving average performance. This shows that addressing stability in the network parametrization is a promising avenue for stabilizing RL agents.
\end{itemize}

\section{Background}

\textbf{Reinforcement Learning.} Reinforcement learning tasks can be formulated as Markov decision processes (MDPs) defined by a state-space $\mathcal{S}$, an action space $\mathcal{A}$, transition probabilities $P$, and a reward distribution $r$ \citep{sutton2018reinforcement}. For continuous control tasks, the action space $\mathcal{A}$ and the state space $\mathcal{S}$ are typically continuous. Each dimension of the action space $\mathcal{A}$  might for example correspond to a joint of a robotic arm, and the space is often bounded to account for physical limitations, e.g., how much force the robotic arm can exert. A common setting is simply $\mathcal{A} = [-1, 1]^{dim(\mathcal{A})}$. At every time step $t$, the agent is in some state $\mathbf{s}_t\! \in\! \mathcal{S}$ and takes an action $\mathbf{a}_t \!\in\! \mathcal{A}$ to arrive at a new state $\mathbf{s}_{t+1} \in \mathcal{S}$. The transition between states given an action is random with transition probability $P : \mathcal{S} \times \mathcal{S} \times \mathcal{A} \rightarrow [0, \infty)$. Given some action $\mathbf{a}_t$ at timestep $t$, the agent also receives a reward $r_t$ from the reward distribution $r$. When benchmarking RL agents, it is common to measure performance by the cumulative rewards $\sum_t r_t$.

\textbf{DDPG.} Deep Deterministic Policy Gradients (DDPG)~\citep{lillicrap2015continuous} is an algorithm for continuous control with roots in Q-learning \citep{watkins1989learning}. Together with its offshoot soft actor-critic (SAC) \citep{haarnoja2018soft}, DDPG forms the backbone of many state-of-the-art algorithms for continuous control \citep{laskin2020curl, laskin2020reinforcement, kostrikov2020image, yarats2021mastering}. DDPG uses a neural network (the \textit{critic}) to predict the Q-value $Q_\phi(\mathbf{a}_t, \mathbf{s}_t)$ for each state-action pair. The parameters $\phi$ of the critic are trained by minimizing the soft Bellman residual:
\begin{equation}
\label{eq:critic_loss}
\mathbb{E}  \bigg( Q_\phi(\mathbf{a}_t, \mathbf{s}_t) - \big[  r_t + \gamma \mathbb{E} [ \hat{Q}(\mathbf{a}_{t+1}, \mathbf{s}_{t+1}) ]  \big] \bigg)^2.
\end{equation}
Here, $r_t$ is the reward obtained and $\mathbb{E} [ \hat{Q}(\mathbf{a}_{t+1}, \mathbf{s}_{t+1})]$ is the Q-value estimated by the \textit{target} critic --  a network whose weights can e.g. be the exponentially averaged weights of the critic. One can also use multi-step returns for defining the critic loss function \citep{sutton1988learning}. The policy $\pi(\mathbf{s}) = \argmax_{\mathbf{a}} Q_\phi(\mathbf{s}, \mathbf{a})$ is intractible since it requires finding $\argmax$ over a continuous action space. Instead the policy is given by an \textit{actor}-network (with parameters $\theta$), which maps each state $\mathbf{s}_t$ to a distribution $\pi_\theta(\mathbf{s}_t)$ over actions. The action is obtained by first sampling from a diagonal normal distribution where the mean  $\boldsymbol{\mu}_\theta(\mathbf{s})$ is given by the actor-network and the standard deviation $\sigma_t$ might depend on time $t$. Thereafter a $\tanh$ non-linearity is applied elementwise to ensure $\mathbf{a} \in [-1,1]^n$ for $n = {dim(\mathcal{A})}$. This procedure defines the policy distribution, formally we have
\begin{gather}
\label{eq:policy}
\mathbf{a}^{pre}_\theta(\mathbf{s}) = \boldsymbol{\mu}_\theta(\mathbf{s}) + \sigma_t \times \boldsymbol{\epsilon} , \qquad \boldsymbol{\epsilon} \sim \mathcal{N} (0, \mathbf{1}) \\
\label{eq:tanh}
\mathbf{a}_\theta(\mathbf{s}) = \tanh ( \mathbf{a}^{pre}_\theta(\mathbf{s}) )
\end{gather}
By sampling $\boldsymbol{\epsilon}$ we obtain an action $\mathbf{a}_\theta$ from the policy $\pi_\theta(\mathbf{s})$. Since the Q-values correspond to the expected discounted rewards, the actions $\mathbf{a}_\theta$ should maximize $Q_\phi(\mathbf{a}_\theta, \mathbf{s})$. Thus, to obtain a gradient update for the actor one uses the gradient $\nabla_\theta Q_\phi(\mathbf{a}_\theta, \mathbf{s})$, effectively calculating gradients \textit{through} the critic network. In control from pixels, the state $\mathbf{s}$ is typically an image processed by a small convolutional network. Image augmentations can dramatically improve performance in this setting \citep{kostrikov2020image, laskin2020reinforcement}, and has become a standard method during training.

\section{Evaluation Variance}
\label{sec:appn_eval_var}

We will now consider the variance arising from evaluating RL agents on a finite set of episodes. Similar to the bias-variance decomposition, we will see that the variance here can be divided into one part that depends upon the randomness of the algorithm, and one term that depends on the randomness from the finite set of episodes we evaluate on. We let $R$ be a set of training runs and $S$ be a set of evaluation episodes sampled. The performance is measured by

$$
\text{perf} = \frac{1}{|R||S|} \sum_{i \in R} \sum_{j \in S} r_{ij}
$$

Let us calculate the variance of this quantity. We will assume that rewards from different runs are independent, and that rewards from the same training run are independent conditioned on that run. We now have

$$
\text{Var}\bigg( \frac{1}{|R||S|}  \sum_{i \in R}  \sum_{j \in S} r_{ij} \bigg)
$$

The variance of a sum of independent variables is simply the sum of their variances. Thus we have

$$
\text{Var}\bigg( \frac{1}{|R||S|}  \sum_{i \in R}  \sum_{j \in S} r_{ij} \bigg) = \frac{1}{|R|} \text{Var}\bigg(  \frac{1}{|S|} \sum_{j \in S} r_{ij} \bigg)
$$

Let $\mu = \mathbb{E}[r_{ij}]$. We then have

$$
\text{Var}\bigg( \frac{1}{|S|}  \sum_{j \in S} r_{ij} \bigg) = \mathbb{E}_{alg,samples} \bigg[\bigg( \frac{1}{|S|}  \sum_{j \in S}r_{ij} - \mu \bigg)^2 \bigg]
$$

Here we explicitly calculate the expectation over random training runs (\textit{alg}) and the random episodes sampled (\textit{samples}). We can condition the expectation upon the random training run and then integrate over this expectation. Furthermore, let $\mu_i = \mathbb{E}[r_{ij} | i]$, i.e. the expected value an episode sampled from training run $i$. We then have

$$
\text{Var}\bigg( \frac{1}{|S|}  \sum_{j \in S} r_{ij} \bigg) = \mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[ \bigg( \frac{1}{|S|}  \sum_{j \in S}r_{ij} - \mu \bigg)^2 \bigg| i \bigg]  \bigg]
$$

$$
= \mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[ \bigg( \frac{1}{|S|}  \sum_{j \in S}r_{ij} - \mu_i + \mu_i -  \mu \bigg)^2 \bigg| i \bigg]\bigg]
$$

$$
= \mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[ \bigg( \frac{1}{|S|}  \sum_{j \in S}r_{ij} - \mu_i \bigg)^2 \bigg| i \bigg]
$$

$$
+ 2 \mathbb{E}_{samples} \bigg[ \bigg( \frac{1}{|S|}  \sum_{j \in S}r_{ij} - \mu_i\bigg) \bigg( \mu_i -  \mu \bigg) \bigg| i \bigg]
$$

$$
+ \mathbb{E}_{samples} \bigg[ \bigg(  \mu_i -  \mu \bigg)^2 \bigg| i \bigg]  \bigg]
$$

The middle part vanishes since $\mu_i = \mathbb{E}[r_{ij} | i]$. For the first term, given a fixed training run the terms $r_{ij}$ are independent. This gives us

$$
\mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[ \bigg( \frac{1}{|S|}  \sum_{j \in S}r_{ij} - \mu_i \bigg)^2 \bigg| i \bigg] = \frac{1}{|S|^2}  \mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[ \bigg( \sum_{j \in S}r_{ij} - |S| \mu_i \bigg)^2 \bigg| i \bigg]
$$

$$
= \frac{1}{|S|^2}  \mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[  \sum_{j \in S} \big( r_{ij} - \mu_i \big)^2 \bigg| i \bigg] = \frac{1}{|S|}  \mathbb{E}_{alg} \bigg[ \mathbb{E}_{samples} \bigg[ \big( r_{ij} - \mu_i \big)^2 \bigg| i \bigg]
$$

We then have

\begin{equation}
\label{eq:eval_var}
\text{Var} (\text{perf}) = \frac{1}{|R|} \underbrace{\mathbb{E}_{alg} [(\mu_{alg} - \mu)^2 ]}_{\text{algorithm variance}} + \frac{1}{|R||S|} \underbrace{\mathbb{E}_{alg} \big[ \mathbb{E}_{samples} [(r - \mu_{alg})^2 | i] \big]}_{\text{sample variance}}
\end{equation}

Here $\mu_{alg} = \mu_i$. The first term is what we are interested in measuring. The second term can be large if the environment is noisy and we sample too few episodes. The second term is straightforward to estimate: run the algorithm N times and measure the performance over M samples. For each run, calculate the variance over the samples. Then average this variance over all runs.

In \Cref{tab:eval_variance} we estimate these quantities over ten training runs and ten episodes for the baseline algorithm. As we can see, the environments have very different characteristics. Some have high sample variance and some have very low sample variance. It is common to evaluate with ten training seeds and ten episodes. Can we improve the variance by evaluating on more episodes, say 100? Even for the task with the highest sample noise (reacher), the relative improvement in the standard deviation of the sample variance, i.e.,

\begin{equation}
\label{eq:eval_var_possible_improvement}
\sqrt{\big( \sigma^2_{alg} + \frac{1}{10} \sigma^2_{sample} \big) / \big( \sigma^2_{alg} + \frac{1}{100} \sigma^2_{sample} \big) } \approx 1.195
\end{equation}

is no larger than 20 \%. Thus, we conclude that only minor gains can come from evaluating on more episodes. Other environments outside the five we consider, however, might benefit more from using more samples.

\begin{table}[]
\centering
\caption{Sample variance and algorithm variance as defined in \cref{eq:eval_var}. Some tasks have high variance and others have low. However, sampling more episodes does not decrease variance considerable in the 10 training seeds and 10 episodes setting, see \cref{eq:eval_var_possible_improvement}.}
\label{tab:eval_variance}
\begin{tabular}{llllll}
\toprule
 method & walker walk & hopper hop & finger turn & acrobot & reacher reach   \\
 \midrule
 sample & 1922.1   & 450.6   & 143763.0  & 11182.9  & 176211.9   \\
alg & 80002.6   &  9774.0   & 55215.6   &  5710.9  & 35231.7    \\
\bottomrule
\end{tabular}
\end{table}

\section{Related work}

\textbf{Variance in RL.} RL is by now commonly accepted to suffer from high variance \citep{clary2019let, lynnerup2020survey, henderson2017deep, engstrom2020implementation}. Given this high variance, there is ample work on how to accurately measure performance in RL \citep{colas2019hitchhiker}. \cite{chan2019measuring} proposes several metrics that can measure different notations of reliability in reinforcement learning. \cite{jordan2020evaluating} proposes an evaluation methodology that takes into account the effects of hyperparameter tuning. Recently \cite{agarwal2021deep} proposes interval estimates of performance. Given this plethora of proposed metrics, we simply report average mean and standard deviation which is commonly reported in practice. Our strategy of simply reducing variance in RL algorithms is complementary to developing robust metrics. Beyond RL, there is work on variance in computer vision \cite{bouthillier2019unreproducible} and NLP \citep{melis2017state, lin2021significant}.

\textbf{Stabilizing RL.} Within Q-learning, it is known that high variance Q-estimates can impede learning \citep{fujimoto2018addressing}. Hence, there is a large amount of work on improving the Q-estimates for DQN: averaging weights \citep{nikishin2018improving}, averaging Q-estimators \citep{anschel2017averaged}, using classical variance reduction methods \citep{jia2020variance} and regretted reward \citep{chen2018stabilizing}. These papers focus on Q-value variance instead of performance variance and apply to the discrete DQN setting, distinct from continuous control. In policy learning there are several methods for decreasing variance of gradient estimates \citep{greensmith2004variance, wang2017meta}, however, these do not directly apply to DDPG. On the architectural side, \cite{parisotto2020stabilizing} stabilizes transformers for RL. On the task-specific side, \cite{kumar2019stabilizing} stabilizes learning for offline tasks while \cite{mao2018variance} considers reducing gradient variance for externally driven environments.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/grid_ours_bottom.pdf}
\vspace{-0.4cm}
\caption{The average absolute value of the actions for the combined agent. We show four representative runs of walker task. Compared to the baseline algorithm (\Cref{fig:grid}), the actions do not saturate early in training despite increasing rapidly. This suggests that our methods improve stability.}
\vspace{-0.4cm}
\label{fig:grid_ours_bottom}
\end{figure}

\section{Discussion}

\textbf{Limitations.} We have considered a single algorithm on a single suite of tasks. Our setting uses a state-of-the-art method on a popular benchmark relatively close to real-world applications. Nonetheless, many important settings remain, e.g. model-based algorithms \citep{hafner2020mastering} and discrete environments \citep{bellemare2013arcade}. This paper shows that \textit{it is possible} to decrease variance with simple modifications, although which methods are effective likely depends upon the base algorithm. We leave evaluating how well our modifications transfer to other settings for future work.

\textbf{Conclusion.} We have investigated causes of variance in a popular continuous control setup. Whereas feature learning and network initialization do not appear to be at fault, we observe that learning instability causes significant variance. We show that simply normalizing the features has a stabilizing effect that decreases the variance and also propose several additional fixes. Our experiments demonstrate that the perception that RL has intrinsically high variance might need to be revisited -- in some cases, numerical issues might instead be at fault.

\newpage

\textbf{Ethics Statement.} There are many positive RL applications -- examples include safe autonomous cars, accelerated scientific discovery, and medical testing. Studying causes of high variance, and proposing fixes to this issue, can benefit real-world adoption of RL techniques. Additionally, reducing the variance of RL experiments would aid the research community as reproducibility can be problematic in the field. However, there are also many potentially harmful military applications. Our study is academic in nature and only considers simulated toy environments, but insights from our work could nonetheless be useful in harmful applications. We do not believe that our work differs from other RL work significantly with regard to ethics.

\textbf{Reproducability Statement.} In \Cref{sec:appn} we detail the infrastructure and software used. The perhaps most important component of generating reproducible findings in RL is to use many seeds. To this end, we only consider five environments but evaluated these thoroughly. The main experiments of the paper (\Cref{tab:main_results}) use 40 seeds whereas all others use 10 seeds.

\textbf{Acknowledgement.} This research is supported in part by the grants from the National Science Foundation (III-1618134, III-1526012, IIS1149882, IIS-1724282, and TRIPODS- 1740822), the Office of Naval Research DOD (N00014- 17-1-2175), Bill and Melinda Gates Foundation. We are thankful for generous support by Zillow and SAP America Inc. This material is based upon work supported by the National Science Foundation under Grant Number CCF-1522054. We are also grateful for support through grant AFOSR-MURI (FA9550-18-1-0136). Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors. We thank Rich Bernstein, Wenting Zhao and Ziwei Liu for help with the manuscript.

\section{Benchmarking Variance}

\subsection{Experimental Setup}

There is a large number of available RL agents and environments to consider. To limit the papers the scope and computational footprint, we consider continuous control from pixels with an actor-critic algorithm. Continuous control from pixels is an environment type close to real-world applications such as autonomous driving \citep{kiran2021deep} and robotics \citep{zhu2020ingredients}. Investigating causes of high variance in this domain could therefore have a large potential impact on downstream applications. We consider the standard continuous control benchmark deepmind control (dm-control) \citep{tassa2020dmcontrol}. Within dm-control, a long line of state-of-the-art models relies on the actor-critic setup \citep{laskin2020curl, laskin2020reinforcement, kostrikov2020image, yarats2021mastering}. We consider DDPG \citep{lillicrap2015continuous}, using the recent data-augmentation based implementation of \cite{yarats2021mastering} which provides state-of-the-art performance and runs fast. We use the default hyperparameters that \cite{yarats2021mastering} uses on the \textit{medium} benchmark (listed in \Cref{sec:appn}) throughout the paper. We will refer to this agent as the \textit{baseline} agent. We are interested in using a large number of random seeds to obtain accurate statistics and thus need to limit the number of environments we consider. To select environments we simply rank the dm-control tasks based upon their relative variance as measured by \cite{yarats2021drqv2}. We select the five tasks with the highest relative variance -- finger turn hard, walker walk, hopper hop, acrobot swingup, and reacher hard (see \Cref{tab:rel_std_choose_tasks} in \Cref{sec:appn} for details). We will refer to these as \textit{environments} or \textit{tasks} interchangeably. For each run, we train the agent for one million frames, or equivalently 1,000 episodes, and evaluate over ten episodes. We use 10 or 40 seeds for each method we consider, these are not shared between experiments, but we reuse the runs for the baseline agent across figures.

\subsection{Verifying Variance}

We first aim to verify that the environments indeed do exhibit high variance. To this end, we run the baseline algorithm with 40 random seeds on each task. \Cref{fig:rnd_individual} shows the result of this experiment for three tasks, see \Cref{sec:appn} for the two other tasks. We show both individual runs as well as mean and standard deviations. The variance is often dominated by \textit{outlier runs} that fail to learn early in the training and never recover. E.g. for the walker task, we see several runs which have not improved during the first 500 episodes, and most of these do not improve in the following 500 episodes either. Except for these outliers, the variance of the walker task is relatively small. We also note that these outliers separate themselves relatively early on, suggesting that the issue that causes high variance arises early in training. If it was possible to eliminate poor outlier runs, we would expect the variance to go down. In the following sections, we will investigate different hypotheses for what might cause these bad runs to fail early.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/rnd_action_and_reseed_corr.pdf}
\caption{(\textbf{Left}) Correlation between runs that share network initialization and starting experience, but otherwise have independent other sources of randomness (e.g. from mini-batch sampling). Across all tasks, the correlation is small. This suggests that initialization and starting experience have limited effect on the outcomes. (\textbf{Right}) The average reward for the walker task when interpolating between a learned policy (random action probability = 0) and random policy (random action probability = 1). The rewards obtained decrease gracefully as we adopt a more random policy. This suggests that the environment is relatively forgiving with respect to actions.}
\label{fig:reseed_corr}
\end{figure}

\subsection{Does initialization matter?}

There are multiple sources of randomness specific to the early training iterations. The agent networks are randomly initialized and the agent is given some initial experience, collected by random actions, to start training on. Could these two initial factors be the cause of the high variance?  To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience, but otherwise have independent randomness (from e.g. the environment, action sampling, and mini-batch sampling). Across all tasks, we repeat this process for ten combinations of initializations and initial experience, from each such combination we start two runs $A, B$ which do not share other sources of randomness. For each time-step $t$ we  measure the Pearson correlation between $score(A)$ and $score(B)$. This correlation, averaged across all timesteps, is shown in \Cref{fig:reseed_corr} (left). We see that the correlation is very small, suggesting that the initialization and initial experience does not have a large effect on the outcome. Specifically, the hypothesis that there are good and bad initializations that leads to good or bad performance is inconsistent with the results of \Cref{fig:reseed_corr} (left). Correlation across time is noisy but centers around zero, see \Cref{sec:appn}.

\subsection{Are the Environments too Sensitive?}

Another cause of variance is the environments. Perhaps the environments require very specific policies to yield good rewards, and unless the agent stumbles upon a good policy early on, learning fails. To study the sensitivity of the environments we consider gradually exchanging a learned policy with a random policy. Specifically, at every timestep $t$, with probability $p$, we exchange the policy's action with a random action. The effect of this, for ten final policies learned for the walker environment, is shown in \Cref{fig:reseed_corr} (right). We see that performance degrades gracefully as we gradually move to a random policy. This suggests that the environment is relatively stable with regard to actions. Results for other environments are similar and are shown in \Cref{sec:appn}. Furthermore, in \Cref{sec:appn_eval_var} we investigate how much variance comes from evaluating the agent on a finite number of episodes. When using ten seeds and ten episodes (the lowest numbers we use), the evaluation variance is dominated by the variance of the random training process. Thus, environment sensitivity and evaluation methodology do not appear to explain the excessive variance.

\subsection{Is feature learning to blame?}

In the early stages of training, the image encoder will extract poor features. Without good representations, it is hard to learn a good policy, and without a policy generating interesting behavior it is hard to learn good representations. If such a vicious cycle was causing runs to fail, jump-starting the encoder to extract good features should help to decrease the variance. To test this hypothesis, we consider initializing the weights of the convolutional encoder to some ``good" values. Specifically, for each task, we pick a run with above-average final performance and extract the convolutional weights at episode 500. The encoder is initialized with these good weights but then trained as normal. We refer to this strategy as warm start. In addition to this fix, we also consider improving the features with contrastive learning methods during early training \citep{laskin2020curl}. We use the loss function proposed in \cite{grill2020bootstrap} which has been shown to be useful in smaller batch sizes (we use 256) and has already successfully been applied in RL \citep{schwarzer2020data}. We use standard methods in contrastive learning: a target network with exponentially moving weights and a \textit{head} network. See \Cref{sec:appn} for details. We consider adding the contrastive loss to the first 10,000 and 20,000 steps and refer to these modifications as contrastive 10k and contrastive 20k. The results for this experiment are shown in \Cref{tab:add_cpc}. For these fixes, the large variance persists even though the performance improves marginally for some environments. We thus hypothesize that representation learning is not the main issue. Indeed, experiments in later sections of the paper show that one can significantly decrease variance without improving representation learning, suggesting that feature quality is not the bottleneck.

\begin{table}[]
\centering
\caption{Performance when using contrastive learning or warm starting the convolutional encoder with good weights. Average reward sometimes improves but variance remain high. This suggests that poor features might not cause the high variance. Indeed, \Cref{tab:ideas} shows that performance can be improved without better representation learning. Metrics are calculated over 10 seeds.}
\label{tab:add_cpc}
\begin{tabular}{lllllll}
\toprule
metric &  method & acrobot  & finger turn  & hopper hop & reacher hard  & walker walk\\
\midrule
$\mu$ & baseline & 122.0  & 278.6  & 176.2 & 678.0 & 769.1 \\
$\mu$ & contrastive 10k & 141.4  & 456.2  & 176.5 & 671.6 & 677.1 \\
$\mu$ & contrastive 20k & 159.4  & 325.3  & 209.5 & 659.6 & 773.8 \\
$\mu$ & warm start & 160.0  & 303.7  & 153.3 & 620.8 & 581.4 \\
\midrule
$\sigma$ & baseline & 76.0  & 221.2  & 90.3 & 167.3 & 350.1 \\
$\sigma$ & contrastive 10k & 68.4  & 276.1  & 104.6 & 222.4 & 425.9 \\
$\sigma$ & contrastive 20k & 67.7  & 191.9  & 71.6 & 164.8 & 374.4 \\
$\sigma$ & warm start & 39.8  & 146.7  & 86.8 & 168.7 & 429.1 \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[b!]
\centering
\includegraphics[width=\textwidth]{figures/grid.pdf}
\caption{Each row shows a different quantity during training for the baseline agent. Row 1 shows rewards, row 2 shows actor gradient norm $\| \nabla \ell \| $, row 3 shows actor weight norm $\| w \|$ and row 4 shows average absolute action $|\mathbf{a}|$. Each column corresponds to an independent run of the walker task: run 1 is successful, runs 2 and 3 eventually learn, while run 4 fails to learn. Before learning starts, all runs suffer from small gradients and weights which barely change. During this period, the average absolute value of actions is close to 1. This causes small gradients since the gradient of $\tanh(x)$ is small when $|\tanh(x)| \approx 1$. This suggests that numerical issues may hamper learning.}
\label{fig:grid}
\end{figure}

\section{Improving stability}

We have considered the effects of initialization, feature learning, and environment sensitivity and found that none of these factors satisfactorily explain the observed variance. Another factor to consider is simply the numerical stability of the learning algorithms. Such learning instability could make the learning process ``chaotic", which would cause high variance. To study this hypothesis we investigate the failure modes of individual runs. In \Cref{fig:grid} we consider some runs that fail as well as some successful runs for the walker task (which we will use as a running example). Specifically, we plot four quantities during training: rewards, actor gradient norm $\| \nabla \ell\|$, actor weight norm $\| w \|$, and average absolute value of actions $|\mathbf{a}|$. All agents initially have poor rewards, but once learning starts, the performance improves relatively steadily. During this initial period of no or little learning, the gradients of the actor-network (row two in \Cref{fig:grid}) are small. Similarly, the weight norm of the actor seems to be constant (row three in \Cref{fig:grid}). This suggests that the actor isn't changing its weight much (technically the weights could be ``rotating", but this seems unlikely). Why would the actor weights not change? Recall that the actor is optimized by taking gradients:
\begin{equation}
\nabla_\theta Q_\phi (\mathbf{s}, \mathbf{a}_\theta(\mathbf{s})) = \sum_i \frac{\partial Q_\phi}{\partial \mathbf{a}_i} \frac{\partial \mathbf{a}_i}{\partial \theta}
\end{equation}
If this gradient is close to zero, we might suspect that $\frac{\partial \mathbf{a}_i}{\partial \theta} \approx 0$. This could easily happen if the $\tanh$ non-linearity in \eqref{eq:tanh} is saturated. In row four of \Cref{fig:grid} we show the average absolute value of the actions, which are all bound to lie in $[-1, 1]$. We see that for the runs that fail the average absolute action stays around 1 throughout training, which implies that the gradients for the actor are small since $|\tanh(x)| \approx 1$ implies $\frac{\partial}{\partial x}\tanh(x) \approx 0$. The cause of saturating tanh must be exploding activations, i.e. $\mathbf{a}^{pre}_\theta(\mathbf{s})$ in \eqref{eq:policy} becoming too large, which saturates the tanh non-linearity. Exploding activations is a known issue in deep learning \citep{bjorck2018understanding, glorot2010understanding}. In fact, all runs seem to initially suffer from saturating tanh, but most runs escape from this poor policy. The Q-values could of course also suffer from instability from exploding activations, although for the walker task the issue seems to mostly come from the policy. These experiments suggest that improving numerical stability and avoiding exploding activations could eliminate failing runs, which in turn would decrease the variance.

\begin{table}[b!]
\centering
\caption{Performance for various stability improvements. These methods generally decrease variance, especially for the walker task where variance decreases by orders of magnitude. Larger stability often leads to higher reward, likely as poor outlier runs are eliminated. Normalizing features gives consistent improvements. Metrics are calculated over 10 seeds.}
\label{tab:ideas}
\begin{tabular}{lllllll}
\toprule
metric &  method & acrobot & finger turn & hopper hop & reacher hard & walker walk\\
\midrule
$\mu$ & baseline & 122.0  & 278.6  & 176.2 & 678.0 & 769.1 \\
$\mu$ & penalty & 128.0  & 431.6  & 241.7 & 584.6 & 956.3 \\
$\mu$ & actor norm & 117.8  & 372.7  & 240.5 & 701.7 & 956.7 \\
$\mu$ & actor+critic norm & 192.2  & 605.8  & 188.7 & 892.7 & 961.5 \\
$\mu$ & layer norm & 43.5  & 661.2  & 218.6 & 659.7 & 952.4 \\
$\mu$ & lr-warmup & 150.3  & 265.4  & 173.7 & 660.9 & 670.4 \\
$\mu$ & grad clip 1 & 138.2  & 214.0  & 206.0 & 520.0 & 675.6 \\
$\mu$ & grad clip 10 & 79.3  & 166.2  & 157.7 & 548.4 & 678.7 \\
\midrule
$\sigma$ & baseline & 76.0  & 221.2  & 90.3 & 167.3 & 350.1 \\
$\sigma$ & penalty & 42.1  & 160.6  & 46.5 & 236.8 & 15.4 \\
$\sigma$ & actor norm & 39.9  & 227.9  & 42.7 & 101.9 & 8.0 \\
$\sigma$ & actor+critic norm & 60.1  & 193.4  & 78.4 & 122.5 & 7.0 \\
$\sigma$ & layer norm & 57.2  & 228.4  & 112.5 & 297.1 & 13.5 \\
$\sigma$ & lr warmup & 53.8  & 133.6  & 98.2 & 133.4 & 423.4 \\
$\sigma$ & grad clip 1 & 60.6  & 127.1  & 92.4 & 231.9 & 425.3 \\
$\sigma$ & grad clip 10 & 56.1  & 143.9  & 113.6 & 243.2 & 426.3 \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Testing Stabilization Methods}

We now consider a few ideas for improving the numerical stability and addressing exploding activations, hoping that these will decrease the variance. We consider four methods:

\textbf{Normalizing activations.} Recall that the actor selects actions from the state using $\boldsymbol{\mu}_\theta(\mathbf{s})$, where $\boldsymbol{\mu}_\theta(\mathbf{s})$ is a deep neural network. \Cref{fig:grid} shows that learning can fail due to exploding activations. To avoid this we propose to parametrize $\boldsymbol{\mu}_\theta(\mathbf{s})$ as a linear function of features with a fixed magnitude. Specifically, let $\Lambda_\theta(\mathbf{s})$ be the penultimate features of the actor-network. We propose to simply normalize $\Lambda^n_\theta (\mathbf{s}) = \Lambda_\theta(\mathbf{s}) /  \| \Lambda_\theta (\mathbf{s})\|$ and let $\boldsymbol{\mu}$ be a linear function of $\Lambda^n_\theta$, i.e. $\boldsymbol{\mu}_\theta(\mathbf{s})  = L \Lambda^n_\theta(\mathbf{s}) $ for a linear function $L$. We refer to this straightforward modification as \textit{actor norm}. Our motivation has been to stabilize the actor, but the critic might similarly benefit from normalizing penultimate features. We refer to the strategy of normalizing both the actor and the critic as \textit{actor+critic norm}. We also consider \textit{layer norm} \citep{ba2016layer} applied similarly to both the actor and critic.

\textbf{Penalizing saturating actions.} Another strategy for avoiding actions with large magnitude is to add a term to the loss function which directly penalizes large actions. One can imagine many such loss functions, we here consider the simple and natural choice of squared loss:
\begin{equation}
\label{eq:pretanh_penalty}
\ell(\theta, \mathbf{s}) = \lambda \| \mathbf{a}^{pre}_\theta(\mathbf{s}) \|^2
\end{equation}
We can simply add this to the loss function times some hyperparameter $\lambda$ and then train the network as normal. By tuning $\lambda$, one should be able to remove the saturating actions illustrated in \Cref{fig:grid}. We heuristically find that $\lambda=0.000001$ works well in practice (see \Cref{sec:appn}) and use this value. We refer to this modification as \textit{penalty}.

\textbf{Learning rate warmup.} One common strategy to improve stability is to start the learning rate at zero and increase it linearly to its top value during early training \citep{goyal2017accurate}. This can improve numerical stability \citep{ma2019adequacy}. We consider using this strategy and linearly increase the learning rate from zero over the first 10,000 steps. We refer to this strategy as \textit{lr warmup}.

\textbf{Gradient clipping.} Another stabilizing strategy is gradient clipping \citep{zhang2019gradient}. To avoid occasional exploding gradients, one simply clips the norm of gradients that are above some threshold. This strategy is popular in NLP \citep{gehring2017convolutional, peters2018deep} and is sometimes used in RL \citep{stable-baselines3}. We consider clipping the gradient when the norm, calculated independently for the actor, critic, and convolutional encoder, is larger than 1 or 10. We refer to these strategies as \textit{grad clip 1} and \textit{grad clip 10}.

\textbf{Results.} In \Cref{tab:ideas} we evaluate the effects of these ideas. We see that it is possible to significantly decrease the variance of the baseline by applying these stabilization methods. Additionally, decreasing the variance increases the average reward -- likely as bad outlier runs are eliminated. However, what methods are suitable for what environment varies. The penalty method has relatively mixed performance. Normalizing activations seems to stably improve performance and decrease rewards, whereas layer norm can make performance worse. The performance improvement coming from normalizing the critic suggests that the critic network also suffers from instabilities. Gradient clipping and learning rate warmup perform poorly in comparison. We conclude that feature normalization seems to be the most performant fix, and note that it requires no hyperparameters.

\subsection{Combining Fixes}

We have found that many of the proposed methods can decrease variance and improve average reward. We now consider combining several methods -- actor and critic penultimate normalization, pre-tanh penalty, and contrastive learning for the first 10,000 steps -- to achieve the best possible performance. In \Cref{tab:main_results} we compare the performance between the baseline and an agent using these three modifications (which we refer to as \textit{combined}) over 40 seeds. We see clearly that the variances go down while the average score goes up. In fact, for the walker task, the variance decreases by orders of magnitude. Additionally, eliminating poor outlier runs benefits the average performance in addition to decreasing variance. While our methods have mostly been designed to help outlier runs that fail catastrophically, they likely help more normal runs too. Learning curves for the combined agent are given in \Cref{sec:appn}. Our experiment demonstrates that the perceived variance in RL is not necessarily inherent to the problem definition and may be addressed with minor modifications.

\begin{table}[]
\centering
\caption{Performance when combining three improvements: penultimate feature normalization, action penalty and early contrastive learning. The variance decreases and the performance improves across all tasks. This demonstrates that one can substantially improve variance in RL experiments by simply improving numerical stability. Metrics are calculated over 40 seeds.}
\label{tab:main_results}
\begin{tabular}{lllllll}
\toprule
metric &  method & acrobot & finger turn & hopper hop & reacher hard & walker walk\\
\midrule
$\mu$ & baseline & 122.0  & 278.6  & 176.2 & 678.0 & 769.1 \\
$\mu$ & combined & \textbf{227.7}  & \textbf{653.5}  & \textbf{231.1} & \textbf{893.4} & \textbf{964.2} \\
\midrule
$\sigma$ & baseline & 76.0  & 221.2  & 90.3 & 167.3 & 350.1 \\
$\sigma$ & combined & \textbf{59.0}  & \textbf{183.1}  & \textbf{40.8} & \textbf{120.0} & \textbf{6.2} \\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[b!]
\centering
\vspace{-0.4cm}
\caption{Performance when changing learning rates. The baseline algorithm fails with larger learning rates whereas our combined agent succeeds. This suggests that our fixes improve the stability. Metrics are calculated over 10 seeds on the walker task.}
\label{tab:lr_stability}
\begin{tabular}{llllll}
\toprule
metric &  method & \num{1e-4}  & \num{3e-4}  & \num{1e-3}  & \num{3e-3}   \\
\midrule
$\mu$ & baseline & 769.1  & 330.2   & 26.8   & 25.5   \\
$\mu$ & combined & 964.2  & 962.2   & 954.9  & 960.1 \\
\midrule
$\sigma$ & baseline & 350.1 & 415.3   &  3.9  &  6.4   \\
$\sigma$ & combined & 6.2  &  4.7   &  5.6  & 5.1  \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Learning rate stability}

To verify that our fixes indeed increase the stability of learning we now consider changing the learning rate. A high learning rate can cause divergence \citep{li2019towards}, and can thus serve as a proxy for learning stability. We consider learning rates in the set $\{ \num{1e-4}, \num{3e-4}, \num{1e-3}, \num{3e-3}  \}$, where $ \num{1e-4}$ is the default learning rate used by \cite{yarats2021mastering}. Results for the walker task are shown in \Cref{tab:lr_stability}. We see that the baseline algorithm fails for large learning rates, the average rewards drop quickly while variance decreases when no effective policy is learned. With our modifications, the agent can tolerate much higher learning rates while retaining good average rewards and small standard deviation. Learning rate is an important hyperparameter, and it might be possible to further improve performance by re-tuning this parameter after our fixes are added. Finally, to verify that the actions do not saturate when using our fixes, we repeat the experiments of \Cref{fig:grid} with our combined agent. The actions are shown in \Cref{fig:grid_ours_bottom}, where we see that the action magnitude indeed goes up quickly early in training but never saturates. This supports our hypothesis that normalizing activations can stabilize training. Further quantities are plotted in \Cref{sec:appn}.

\section{Appendix}
\label{sec:appn}

\textbf{Tuning $\lambda$.} For tuning the hyperparameters $\lambda$ defined in \eqref{eq:pretanh_penalty} we search over the set $\{\num{1e-1} , \num{1e-2} ,\num{1e-3} ,\num{1e-4} ,\num{1e-5}, \num{1e-6}\}$. For each value, we run the hopper task for a short amount of time with five different seeds. We found that too large $\lambda$ penalized actions too much, and that even very small values like $\num{1e-6}$ sufficed to give improvement.

\textbf{Details on Contrastive Learning.} We extract features from the convolutional network and the first feedforward layer of the critic, which projects the feature map to 50-dimensions. Feature vectors are normalized to unit norm before being fed into the contrastive loss \citep{bachman2019learning}. We use exponentially moving weights for the target contrastive network \citep{he2019moco}, using the same decay parameters as for the target critic. We use a \textit{head} for the online network which is simply a two-layer MLP with hidden dimension 512 and final dimension 50 \citep{chen2020simple}. The two augmentations we consider are simply two adjacent frames with augmentations that the DDPG agent of \cite{yarats2021mastering} considers naturally. We use the square loss as proposed in \cite{grill2020bootstrap}, which has been shown to be useful in smaller batch sizes. The contrastive loss is then simply added to the normal loss function for the critic.

\textbf{Infrastructure.} We run our experiments on Nvidia Tesla V100 GPUs and Intel Xeon CPUs. The GPUs use CUDA 11.1 and CUDNN 8.0.0.5. We use PyTorch 1.9.0 and python 3.8.10. Our experiments are all based upon the open-source implementation of \cite{yarats2021mastering}.

\begin{table}[h]
\centering
\caption{Variance across tasks as reported by \cite{yarats2021mastering}. Mean reward and task variance often increase in tandem -- to avoid bias towards easy tasks with high mean and thus high variance, we rank tasks based upon relative variance $\sigma / \mu$. We use the five tasks with the highest relative variance, removing one duplicate: finger turn easy/hard. Used tasks are highlighted. Scores are measured just before 1 million frames (i.e. frame 980000) since not all runs have scores at the 1 million mark.  All numbers are rounded to two decimal digits.}
\label{tab:rel_std_choose_tasks}
\begin{tabular}{llll}
\toprule
task & $\sigma / \mu$ & $\mu$ & $\sigma$ \\
\midrule
\cellcolor{blue!25}finger turn hard &  0.74 &  268.90 & 198.36 \\
\cellcolor{blue!25}walker walk &  0.48 &  769.66 & 371.48 \\
\cellcolor{blue!25}hopper hop &  0.36 &  221.13 & 80.63 \\
finger turn easy &  0.31 &  558.52 & 174.19 \\
\cellcolor{blue!25}acrobot swingup &  0.31 &  177.65 & 54.77 \\
\cellcolor{blue!25}reacher hard &  0.22 &  628.29 & 135.57 \\
walker run &  0.20 &  538.59 & 110.02 \\
cup catch &  0.13 &  909.95 & 118.06 \\
finger spin &  0.12 &  860.30 & 104.18 \\
quadruped run &  0.12 &  446.49 & 53.51 \\
quadruped walk &  0.12 &  732.52 & 87.19 \\
cartpole balance sparse &  0.12 &  962.30 & 113.10 \\
reach duplo &  0.08 &  202.51 & 16.99 \\
cartpole swingup sparse &  0.08 &  760.11 & 60.11 \\
cheetah run &  0.07 &  710.24 & 52.43 \\
reacher easy &  0.05 &  931.54 & 50.86 \\
pendulum swingup &  0.04 &  838.49 & 36.38 \\
hopper stand &  0.02 &  917.85 & 20.45 \\
cartpole swingup &  0.02 &  864.68 & 15.47 \\
walker stand &  0.01 &  980.72 & 5.21 \\
cartpole balance &  0.01 &  993.45 & 5.00 \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{figures/rnd_individual_appn.pdf}
\caption{Learning curves for the two tasks not shown in \Cref{fig:rnd_individual}. The upper row shows the mean reward with one standard deviation indicated by shading. The bottom row show learning curves for individual runs. Although the signal is noisy, several outlier runs fail to learn. }
\label{fig:rnd_individ_appn}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{figures/rnd_action.pdf}
\caption{Rewards obtained when gradually replacing a learned policy with a random policy. At every time-step $t$, the action suggested by the learned policy is replaced with a random action with probability $p$. We start from the final policy learned by the baseline algorithm and average over ten starting policies. Across all tasks, the rewards decrease gracefully with a more random policy, suggesting that the environments are robust to perturbed policies.}
\label{fig:rnd_action}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/corr_over_time.pdf}
\caption{Correlation between the evaluation scores of two runs that share network initialization and seed experience, but otherwise have different randomness. Correlation is calculated across ten runs and plotted against time. The correlation is noisy for many environments but oscillates around zero, suggesting that there is little actual correlation and that network initialization and seed experience have a small effect.}
\label{fig:corr_over_time}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/grid_ours.pdf}
\caption{Four quantities shown during training for the combined agent. We show reward, norm of actor gradient $\| \nabla \ell \|$, norm of actor weight $\|w \|$ and average absolute action $| \mathbf{a} |$. The actions never saturate and learning starts quickly. Compare with \Cref{fig:grid}, which shows the same quantities for the baseline algorithm.}
\label{fig:grid_ours}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/individual_ours.pdf}
\caption{Learning curves when using our combined agent. We show mean reward with one standard deviation indicated by shading and learning curves for individual runs.}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{figures/train_rewards.pdf}
\caption{Rewards obtained during training for the runs illustrated in \Cref{fig:grid}. Note that the rewards are nonzero from the start of training. This implies that the agent observes rewards and can train on these. Thus, lack of rewards is not the cause of the poor performance.}
\end{figure}

\begin{table}
\caption{Hyperparameters used throughout the paper. These follow \cite{yarats2021mastering} for the \textit{medium} tasks.}
\centering
\begin{tabular}{lc}
\toprule
parameter        & value \\
\midrule
learning rate & \num{1e-4} \\
soft-update $\tau$ & \num{1e-2} \\
update frequency & $2$ \\
stddev. schedule &  linear(1.0, 0.1, 500000) \\
stddev. clip & $0.3$ \\
action repeat & $2$ \\
seed frames & \num{4e3} \\
batch size & $256$ \\
$n$-step returns & $3$ \\
discount $\gamma$ & $0.99$ \\
\bottomrule
\end{tabular}
\end{table}

\clearpage

\input{text/eval_variance_small}

redpajama_set_name: RedPajamaArXiv


text:
Grounds Open 24/7/365
Website User Privacy and Policies
Explore by Day
Waterfront Event Center
The Guest House at the Headlands
15675 Headlands Rd,
Mackinaw City, MI
headlands@emmetcounty.org
Headlands International Dark Sky ParkPrograms & Events
Dark Sky Park Program Coordinator
Welcome to the Headlands International Dark Sky Park! The grounds, trails, and viewing areas at Headlands are open 24 hours a day, every day. Visitors are welcome to stay out through the night for dark-sky viewing opportunities, but camping is not permitted. Units like tents and campers are not allowed in the park. The Headlands is not intended as an overnight sleeping destination but instead is designed as a place to stay awake and view the stars. You may bring blankets, sleeping bags, chairs, food, beverages, etc. When packing, keep in mind that temperatures are typically 10 degrees lower than expected due to our proximity to the lake shore.
To protect the darkness of the park, please use red-filtered flashlights during your visit to the Headlands. Learn more here.
Our programs take place rain or shine, and no reservations are required unless otherwise noted.
Please save some time during your visit to stop by our "Out of This World" Gift Shop!
The Observatory is limited to park staff and researchers. Visuals when the Observatory is open are projected onto the big screen monitors on the main level. Professional star-gazers and astronomers are available on site to enhance your viewing experience during scheduled observing nights.
While the grounds, trails, viewing areas and restrooms at Headlands are always open and freely accessible, the Waterfront Event Center is only open to the public during scheduled programming and gift shop hours. The Waterfront Event Center at the Headlands is available for private rental.
To stay up-to-date on news and events at the Headlands International Dark Sky Park, register for email blasts by contacting scox@emmetcounty.org and be sure to follow our Facebook page for more news and photos.
Ash Dome aurora autumn equinox chinese new year dark sky Dark Sky Park dark sky park programs Delta Aquarid Dr. Good Hart eclipses 2017 Egypt Emmet County equinox fireworks Guest House Hafiz Halloween Harvest Moon Headlands Headlands Challenge International Dark Sky Association International Dark Sky Week Jill Jack John Hill Photography July stargazing Jupiter Lake Michigan lantern walk Leonid libra lights out Lunt Lyrid Mackinac Mackinaw Mackinaw City meteor shower meteor showers new year night sky photography northern lights October Orion Orionid Osiris Perseid Meteor Shower Perseids Planewave Pleiades Roger McCormick rooster Rumi Saturn seasonal festival shoreline solar eclipse Solstice Staewide Astronomy Night sufi poetry SWAN Temple-Tuttle Triple fright night trivia Valentine's Day Waterfront Center Winter Solstice Yoga
August 2016 – March 2017 Aug 2016 – Mar 2017
New in 2016: Starlight Trivia on Friday nights! @ Shoreline Dark Sky Viewing Area
Friday night trivia tests your knowledge of the stars ~ photo by Aliana Lee
Fridays throughout the summer, starting July 1 (and unless there is a formal program scheduled) join us at the shoreline viewing area with your favorite team of family and friends for Starlight Trivia, designed to test and delight your knowledge of the stars! In the event of inclement weather, Starlight Trivia will take place at the Heritage Village Pavilion, across the street from Headlands main entrance. There are prizes to win, star names to learn, and plenty of fun to be had!
Remaining Starlight Trivia dates are July 29; and Aug. 5, 19, 26, and trivia will conclude with one last hoorah on Sept. 2.
Tags: Dark Sky Park Headlands Lake Michigan Mackinac Mackinaw shoreline trivia
Harvest Moon Dance with the Jill Jack Band! @ Festival Tent near the Headlands entrance
Sep 17 @ 7:30 pm – 10:00 pm
Harvest Moon is the Full Moon closest to Equinox each year, and the Headlands International Dark Sky Park in Mackinaw City is planning a dance to celebrate the change of the season.
The Harvest Moon Dance will take place Saturday night, Sept. 17, under the festival tent with live music featuring the Jill Jack Band, of Detroit. Jill and her players will set the mood with their performance beginning at 7:30 p.m., followed by sunset at 7:47 p.m. and Harvest Moon rising at 8:30 p.m. The program is free and no reservations are necessary, but come early to get your spot on the dance floor. The program is scheduled until 10 p.m. The Jill Jack Band is known for entertaining with strong audience connection and their high-energy rock and blues that can segue into original folk ballads, demonstrating why Jill has won numerous Detroit Music Awards, including 2015's Outstanding American Songwriter Award.
While most evenings visitors to the park must walk a mile in to the dark sky viewing area along the shoreline, on this evening a tent will be placed near the Headlands entrance.
"Jill is an exceptional talent," said Mary Stewart Adams, Headlands Dark Sky Park Program Director, quoting from a recent Detroit Free Press article which reported that "her voice can soar like opera and smoke like the blues … one of the strongest voices in Detroit, or in the country for that matter."
"Harvest Moon is all about celebrating the bounty of the seasons and the work of cultivating a healthy harmony in our lives and the things we must tend, " added Adams. "Of all the Moon's phases, Full Moon is like the release point, when everything we're gathering up in the cycle is let go, so the natural world can have its turn. When it comes to Harvest Moon, we dance, to celebrate everything that goes into a success and challenge of getting to a harmonious experience of nature and its seasons."
And what a season it's been at the Headlands. With construction of the new Waterfront Event Center still restricting parking on site at Headlands, guests all season long have trekked one mile through the woods to the shoreline viewing area where falling stars, the Milky Way, and the elusive Northern Lights have held imaginations with wonder, Adams noted.
"I'm particularly looking forward to this event because it's a great way to show our gratitude for the goodwill of everyone who came out this season," said Adams. "We welcomed thousands and thousands of guests to the Headlands; it's been another amazing summer under the night sky in Emmet County."
Tags: Harvest Moon Headlands Jill Jack
8th annual Headlands Challenge and Triple Fright Night details @ Park Entrance
Oct 29 @ 8:00 pm – 10:00 pm
Dare the dark at the Headlands Challenge on Saturday, Oct. 29, 2016!
The spookiest time of year can be the most beautiful, too! Emmet County will host the eighth annual Headlands Challenge event at the Headlands International Dark Sky Park in Mackinaw City, on Saturday, Oct. 29, from 8 to 10 pm, during which time guests are dared to walk the 1-mile, dimly lighted path from the Headlands entrance to the Guest House, or take the unlit woods route – shorter, but darker, and not for the faint of heart. Both end at the Guest House, where you can find your fortune and hear seasonal storytelling under the stars. Light (and limited) refreshments will also be part of this free, family-friendly event that was the first dark sky event ever held on the Headlands property.
It all started one stormy October when we joined night sky enthusiasts around the world to celebrate the 400th anniversary of Galileo first using a telescope to look at the night sky. Rather than just rely on telescopes, here at Headlands we used the spooky mood of the season and storytelling to celebrate the night, and won commendation for our efforts from the International Astronomers Union Office of Astronomy Development.
The fun continues every year at this time, and has developed to include trick-or-treating at Heritage Village, just across the road from the Headlands entrance, from 6 to 8 p.m. and the haunted McGulpin Point Lighthouse, open until 11 p.m. All events are free and open to the public, with no RSVP required. Trolley service between Heritage Village and the lighthouse will be provided at no charge, beginning at 6 p.m. at Heritage Village, where parking will be available. The trolley will not service the Headlands, as the Challenge is strictly a walking event.
"Our free monthly Dark Sky Park programs are intended to get area residents and visitors out into the dark to observe the amazing night sky we have protected over Emmet County and Northern Michigan," said Mary Stewart Adams, Dark Sky Park Program Director at the Headlands. "The Headlands Challenge is an especially fun event because it really celebrates the natural mood of the season, which is the cross quarter time of moving from greater daylight to greater darkness." Sunset on Oct. 29 is at 6:30 p.m., and this night will be especially dark because the Moon is hidden from view, wending its way toward new phase after the event. This means the evening will be deep and dark, Adams noted. Wagons/strollers are suggested for those walking with young children along the 1-mile road.
See you there, if you dare!
In addition to the Headlands Challenge, Emmet County is also hosting special trick-or-treating hours at McGulpin Point Lighthouse, which is located a short distance from the Headlands. McGulpin Point will be decorated in a Wizard of Oz theme with special hours of 9 a.m. to 11 p.m. on Oct. 29; and 9 a.m. to 7 p.m. on Oct. 30-31.
For further information on Headlands Challenge, contact Mary Stewart Adams, Headlands Program Director at 231-348-1713, or Beth Anne Eckerle, Emmet County Director of Communications and Web Development, at (231) 348-1704 or email beckerle@emmetcounty.org.
The three park properties are located about 2 miles west of downtown Mackinaw City.
The Headlands: 15675 Headlands Road, Mackinaw City, MI www.MIdarkskypark.org
McGulpin Point: 500 Headlands Road, Mackinaw City, MI www.mcgulpinpoint.org
Heritage Village: 1425 W. Central Ave., Mackinaw City, MI (use Headlands Road entrance) www.mackinawhistory.org
Tags: Dark Sky Park Halloween Headlands Challenge Mackinaw City October
'Friday Night (No) Lights' for the Leonids @ Shoreline viewing area at Headlands
Nov 18 @ 10:00 pm – 11:45 pm
Lantern walking through November woods at Headlands
We know it's still football season under the big stadium lights of fall, but it's also the peak of the lion's meteor shower and you're invited to join us under the dark of night at Headlands for a beautiful experience under the starry skies.
Program Director Mary Stewart Adams will meet guests at the Headlands entrance and give a guided tour through the beautiful moonlit woods to the viewing area at the shoreline, to catch the falling stars and hear further tales of the night. This is a one mile walk in the woods, and guests are invited to bring lanterns, to coincide with seasonal traditions of taking lantern walks into dark November nights. Please be prepared with walking shoes and layers of warm clothes. The Leonid Meteor Shower peaks overhead at this time, as earth travels through the wake of starry stuff left in the trail of the Comet Temple-Tuttle.
"The Leonid is a really variable meteor shower," said Adams, "but it's also one of the most historically significant, because it gave rise to the science and study of meteor showers when it caused an incredible outburst in the early 1800s. And it's November, when tradition holds that taking a walk by lantern light is done to celebrate the strength of inner light despite the challenge of growing, outer darkness."
The Moon will be Full just a few days prior to the peak of the Leonid Meteor Shower, and while that can diminish views of the less bright meteors, there is still the promise of beautiful stargazing. Orion will be solidly over the horizon in the east, chasing the star cluster of the Pleiades across the sky, and the Andromeda Galaxy will be seen spiraling directly overhead. In addition to providing star maps and a guided walking tour by lantern light, we'll have our telescopes out for peering deeper into the night while we wait for wishing stars to fall through the sky! Participants should dress for low temperatures. "These colder nights make for some great stargazing because there's less haze in the atmosphere, and the dark seems to be more velvety and richer" said Adams.
The radiant of the Leonid Meteor Shower is in the sickle, or head region of the mighty Lion
Tags: Dark Sky Park Headlands lantern walk Leonid Temple-Tuttle
Snowshoe Hike Through the 12 Days of Christmas @ Meet at Park Entrance
The last New Moon of 2016 occurs during the sacred season of dark and quiet that bridges the old year to the new, traditionally known as the 12 Days of Christmas, allowing for unique contemplation of where we've been, and where we're going! You're invited to contemplate this passage of time in the old growth forest of Headlands, and to participate in engaging storytelling of the night sky. The stars of the season have lent themselves to many cultural tales and traditions, some of which will be shared both indoors and out, especially since these 12 days correspond to the 12 months of the year, and these to the 12 constellations of the zodiac. In addition to snowshoe hiking through the woods, guests can participate in making a workbook/journal designed to allow for contemplation of life experience relative to the stars.
"Winter stargazing can be deeply satisfying, because the cool air makes the night crisp and clear," said Mary Stewart Adams, Program Director for the Headlands International Dark Sky Park.
Program happens rain, snow or shine! If we can't take to the trails with our snowshoes, then we'll hike the woods and enjoy the quiet outdoors with tales of the night sky.
Limited number of snowshoes available by reservation, but it is best to bring your own. (231) 348-1713 to reserve a pair.
Plan Your New Year with the Stars @ Guest House Headlands International Dark Sky Park
In ages gone by, community life was centered around the observance of celestial phenomena, with the consequence that the sites from which this observing took place would naturally be the place for celebrating life's most significant moments. At Headlands, we're stepping into the wisdom of this tradition with our first program of the New Year, during which we will highlight all the terrific phenomena coming toward us in 2017, designed to support all of us in our effort to find harmony in the greater rhythm of things! Though not required, we recommend you bring calendars and writing instruments so you can be a scribe of the stars all year long by knowing in advance how to center your year around meteor showers, eclipses, and all the other starry events unfolding overhead. During this program we will provide specific dates and times for the celestial highlights of the year, with viewing tips and the cultural history of the phenomena.
Tags: dark sky park programs eclipses 2017 Guest House Headlands meteor showers
A Winter Moon makes for the Darkest Night of the Year @ Guest House Headlands International Dark Sky Park
The position of the Big Dipper region of Ursa Major can be used like a mighty celestial clock to determine the seasons of the year and the time of night!
January's New Moon carries the promise of deep darkness for finding the stars and constellations that are used to determine the celestial new year, not only in the Chinese Calendar, but in the Native American and Christian cultures as well. Gather at Headlands just before sunset to learn about these different traditions, to craft star calendars, celebrate the year of the rooster, and to follow it all up with some winter stargazing. Some supplies provided, though you are welcome to bring your own. Suitable for ages 9 to 90…
Tags: chinese new year Dark Sky Park Headlands new year Orion Pleiades rooster
The Romance of the Stars @ Guest House Headlands International Dark Sky Park
In February we peer into the opposite sweep of stars in our Milky Way spiral than is pictured here from late summer 2016 at Headlands, which can make the individual stars appear more brilliant. Photo by Joe Garza.
In the heart of winter and the approaching Valentine's Day, Headlands will host special guests, musician Laszlo Slomovitz and poet Jennifer Burd with their extraordinary project of setting the mystic poems of Rumi and Hafiz to song. "From the cultural perspective, poetry is the closest we can come to the stars, and through this program, Laszlo demonstrates the beauty of such an idea," said Program Director Mary Stewart Adams. "Tonight is an ideal date night for couples and individuals young and old, newly in love or celebrating the enduring romance of a fixed-star union." Guests to tonight's program may bring snack and beverage and arrive early to catch sunset at 6 pm, or arrive 6:30-6:45 pm to park, get settled and wait for the show to begin' plan to stay late, because the Moon will touch the heart star Regulus in the midnight sky! "This is love: to fly toward a secret sky, to cause a hundred veils to fall each moment…"~Rumi
Tags: Hafiz Headlands Rumi sufi poetry Valentine's Day
Catch a falling star…on camera with Shawn Malone @ Guest House at Headlands
Shawn Malone is known far and wide for her exceptional skill in capturing the northern lights, as shown here!
Nationally-acclaimed and award-winning night sky and northern lights photographer extraordinaire Shawn Malone of Lake Superior Photo is back by popular demand for an exciting hands-on lecture and workshop at the Headlands Guest House , just as Spring Equinox approaches and the aurora get more active! The program will occur in two parts, with a free lecture open to the public, followed by a hands-on workshop that requires advance registration at 231-348-1713 due to the limited number of spaces available.
Malone says she can't remember a time when she wasn't keen on capturing and recording the world around her through photography. "There's always been a latent interest. Even as a kid I carried around a film instamatic."
Malone turned to the dark side, the dark side of night, that is, to photograph nature at its most elusive and expressive in the late 1990s, when she photographed her first aurora, also known as the Northern Lights.
Since then, Malone has enjoyed an enviable reputation through national media, with award-winning recognition from Smithsonian Magazine, publication in National Geographic, and with her work appearing multiple times on NBC Nightly News. "I consider Smithsonian and National Geographic the two biggest (photography) institutions in the world. Their recognition was humbling."
Malone, who sells her work around the world from her gallery and studio, Lake Superior Photo, in Marquette, MI, will share the ups and downs of this success, and the do's and dont's of night sky photography at the Headlands Guest House during a two-part event that will include a public lecture and a hands-on photography workshop. The workshop portion of the program is by advance registration only.
Malone says that her current focus in photography is on a "wet plate collodion" process with large format cameras. "It's a turn-of-the-century process that preceded film and instead uses chemicals processed on tin or glass. I'm looking forward to getting back to the roots of the photographic process and image that is in a totally different direction from anything I've done in the past, kind of in rebellion against the digital age and lack of authenticity of photography with all the over-the-top manipulation that is occurring these days."
Reservations are not necessary for the lecture portion of Malone's program at the Headlands, but are required for the hands-on workshop that will follow, by calling 231-348-1713 or by email at darksky@emmetcounty.org. Space is limited for the workshop and a suggested donation of $10 is requested.
Tags: aurora night sky photography northern lights
15675 Headlands Rd, Mackinaw City, MI 49701
(231) 427-1001 headlands@emmetcounty.org
© 2015 All Rights Reserved • Produced by Gaslight Media
& Business
Dark Sky Park
& Refuse
Emmet/Charlevoix
Pellston
County EMS
McGulpin
Petosega
Residential Postal Code*

redpajama_set_name: RedPajamaCommonCrawl


Physical Description: 114 p. : ill., map ; 28 cm.
Statement of Responsibility: by Richard Green ... <et al.>.
Bibliography: Includes bibliographical references (p. 72).
Appendix II .......................... ......... ............ 111.
with the underlying Floridan aquifer system.
core-drilling rig, personnel, and split-spoon sampling equipment.
and grain size, and tested for relative hydraulic conductivity.
succeeding sections of the report.
analysis data sheets, and the permeameter calculation forms.
Figure 1: Core location map.
clear plastic liner, was hammer-driven through each interval.
Split-spoon samples were taken in each well until the top of the.
County Study wells and entered in the county well-file data base.
depths, elevations, and locations of the ten wells.
permeameter analysis and then cut open lengthwise with a table saw.
taken from the split-spoon and continuous cores.
induration, cement type, accessory minerals, and fossils.
descriptions of each of the wells described in this study.
*From 7-1/2 minute U. S. G. S. topographic quadrangle maps.
the permeameter tests were preformed.
up, and defines the terms in this formula.
Figure 2: Typical Falling-head Permeamter setup.
and was deemed to be relatively impermeable. If a sample did show.
remain on the permeameter until three tests could be performed.
various soil and rock type.
was drilled for this project.
a beaker and saved for pipette analysis.
sieving was assumed to be the combined silt and clay weight.
saved and added to the beaker which contained the fine fraction.
percentage of sand and silt-clay fraction for each sample.
using the pipette method (Friedman and Johnson, 1982; Folk, 1974).
because the sample fines flocculated.
dispersing agent and then poured into a 1000 ml graduated cylinder.
1982; Folk 1974) the weight in each phi class was calculated.
calculated to determine the clay and silt fractions of each sample.
Table 5 lists the sand, silt, and clay percentages for each sample.
investigate the mineralogy of the ten study wells.
a continuous core, whenever there was a change in lithology.
Therefore, well W-16207 sample intervals are quite variable.
samples. The first approach was to analyze the sample as.a bulklc"
produce oriented samples for XRD.
The results of the XRD analysis are listed in Tables 6 and 7.
identified. A total of eleven minerals were positively identified.
Table 7 contains the results of the clay fraction analyses.
Mineral abundances were determined from the relative peak heights.
phosphates, are derived from the chemical weathering of francolite.
mineral, has tentatively been identified in one sample.
In these two tables of x-ray diffractometer datat., 'he4^"
is present in one sample from well #4 (#4-1).
The letter C implies.that the mineral is common, or abundant.
a major constituent of the sample.
sediments in north Florida and Georgia.
comprised of porous marine limestones of the Eocene Ocala Group.
Locally, the Floridan aquifer system is overlain by var4ngi.r.
access to the aquifer (Macesich, 1988).
surficial aquifer system and the intermediate confining unit.
hillside slope at the northeastern edge of the Kanapaha Prairie.
above the Floridan aquifer system are perforated.
Figure 3: Columnar section for well #1 (W-16198).
.., .. ...., .... ............ .
.45 .'. :; .: ..
Dark yellowish-brown clayey, organic-rich quartz sand.
SGrayish-brown to dark yellowish-brown clayey, organic-rich quartz sand.
Grayish-brown clayey, organic-rich, dolomitic quartz sand.
Grayish brown to dark yellowish-brown clayey, organic-rich quartz sand.
Grayish-brown to very light gray clayey, organic-rich quartz sand.
Grayish-brown to dark yellowish-brown clayey, dolomitic, organic-rich quartz sand.
Yellowish-gray calcareous, phosphatic quartz sand.
Grayish-brown clayey quartz sand with chert fragments and Eocene fossils.
Light blue-green to grayish-brown chart with sand and limestone fragments.
layer developed at the top of the Ocala Group limestone.
between 0 to 30.5 feet bls.
to 12.5 feet bis, 20.5 to 22.5 feet bis, and 30.5 to 32.5 feet bls.
had very low hydraulic conductivity values of 10'8cm/s (Table 4).
feet bls sample contains kaolinite, wavelite, and smectite clays.
dominant minerals present (Table 6).
sample suggest the top of the limestone was reached.
Figure 4: Columnar section for well #2 (W-16199).
Moderate brown to moderate yellowish-brown clayey, organic quartz sand.
Dark yellowish-brown clayey, organic quartz sand.
Dark yellowish-brown sandy, organic clay with chert fragments.
Dark yellowish-brown clayey quartz sand.
Light olive-gray clayey, organic quartz sand.
Light olive-gray clayey, dolomitic, phosphatic quartz sand.
Figure 5: Columnar section for well #3 (W-16200).
Very lIlht-orange clayey quartz sand.
Yellowin lsh-gray to darght yellowish orange clayey, organic quartz sand.
Yellowish-gray to light yellowish-orange clayey quartz a and.
20 Light greenish-yellow sandy, phosphatic clay.
Grayish-brown to yellowish-gray clayey quartz sand.
30 ......... Very lightornge to white clayey, phosphatic quartz sand.
.i..........:.:....: Grayish-brown to yellowish-gray clayey quartz sand.
:40 ...O.. Very light-orange to white clayey, phosphatic quartz sand.
~~ .Grayish brown to dark yellowish-orange clayey quartz sand with silica cement.
-- Yellowish-gray to light olive clayey, phosphatic quartz ind with limestone fragments.
Z r.'"":",:-":":"i::":: "-':" Yellowishgray to Ilght-olive clayey, phosphatic quartz sand with clay and limestone fragments.
50 'r-:--.-.;-. : Very light-orange clayey quartz sand.
...- White to dark yellowllhorange sandy, phosphatic calcilutite.
.i. IYellowish.gray calcareous, phosphetic quartz sand.
6 0 Yellowish-gry sandy, phosphatic calclutite.
.... ..Yellowish-gray to moderate yellowish-brown calcareous, phosphatic quartz sand.
Yellowish-gray to white foslliferous limestone.
L90 oI -- T.D.-90 FEET.
48 percent of the sample. Smectite is the dominant clay present.
clay content, the interval tested as a low conductivity of -cm/s.
of undifferentiated Pleistocene-Holocene clayey, quartz sands.
Figure 6: Columnar section for well #4 (W-16201).
Grayish-orange to yellowish-gray clayey, organic-rich quartz sand.
Light-red to light-brown clayey, organic-rich, dolomitic quartz sand.
Very light-orange clayey quartz sand.
Light-brown clayey, calcareous quartz sand.
Very light-orange to white fossiliferous limestone.
trace amounts of kaolinite comprise the clay component.
common mineral in both the bulk and clay fraction x-ray analyses.
minerals throughout the entire section (Table 6).
in the undifferentiated Pleistocene-Holocene section.
Figure 7: Columnar section for well #5 (W-16202).
Yellowish-rey to moderate yellowish-brown clayey quartz sand with plant remains.
Yellowish-gray to dark yellowish-orange sandy clay with plant remains.
Dark grayish-yellow sandy, phop hatic clay.
Yellowish-gray to grayish orange sandy clay.
White to dark yellowish-orange sandy, phosphatic clay.
Grayish-brown clayey, phosphatle quartz sand.
Grayish-yellow clayey. phosphatic quartz sand.
Yellowish-gray to moderate yellowish-brown clayey quartz sand.
Very light-orange clayey, phosphatic quartz sand.
Grayish-brown to white calcareous, phosphatic quartz sand.
SWhite calcareous, phosphatic quartz sand.
Grayish-orange-pink to dark yellowish-orange calcareous, phosphatic quartz sand.
Light greenish-yellow clay with limestone Iragments.
Very light-orange calcareous, phosphatic quartz sand.
SYellowish-grey calcareous, phosphatic quartz sand with bryozoan fossils.
Pinkish.gray calcareous quartz sand with chert fragments.
Figure 8: Columnar section for well #6 (W-16203).
Light-olive to light brown sandy, phosphatic clay. .--.;.
Grayish-brown to light brown clayey quartz sand.
Very llht-orange to yellowish gray fossiliferous limestone.
suggests a fluvial or lacustrine origin for these sediments.
Figure 9: Columnar section for well #7 (W-16204).
Dark gray to yellowlih-gray sandy peat.
Yellowish-gray calcareous. sandy, phosphatic clay.
Dark-brown calcareous, sandy phosphatic clay.
Dark yellowish-brown to dark-brown sandy, calcareous clay with freshwater gastropods.
Dark yellowish-brown calcareous, sandy clay with freshwater mollusk fragments.
Moderate gray to black organic-rich clay with mollusk shell fragments.
Dark yellowish-brown to black clayey, sandy, organic-rich limestone.
Black to moderate-gray clayey quartz sand with plant remains and limestone.
the Alachua Lake Cross Va lley geomorphic1 zone' (White, 1970).
instead in a moderately-indurated Hawthorn Group cemented sand.
phosphate may be derived from reworked Hawthorn Group sediments.
Four of the split-spoon cores were taken in the Hawthorn Group.
Columnar section for well #8 (W-16205).
Moderate yellowish-brown clayey, organic-rich quartz and.
Yellowiih-gray to grayish-brown clayey quartz sand.
Gray Ih-brown clayey, organic-rich quartz sand.
SVery light-gray to yellowishgray clayey quartz sand.
Grayish-brown to dark yllow shorange sandy clay.
YellowlshWay sandy, clayey, phosphatic doloslit.
Ought olive dolomltic, phosphatic clay.
S Yllowtih-gray to moderate yellowlsh-brown clayey, phosphatic quartz sand.
- White to very light gray clayey, photphatlc. sandy dolomite.
- Modrate orangeplnk clayey quartz snd.
Figure 11: Columnar section for well #9 (W-16206).
BC i 1 I L I ,: ; i.
Grayish-brown to moderate reddish-orange clayey, organic, quartz sand.
SLight olive-gray clayey quartz sand.
Grayish-brown clayey, organic quartz sand.
Light olive-gray clayey quartz sand.
Grayish brown to moderate yellowish-brown clayey, phosphatic quartz sand.
Grayish brown to yellowish-gray sandy, clayey, phosphatic dolosilt.
Light olive-gray to yellowish gray sandy, dolomitic clay.
Light olivegray sandy, clayey, phosphatic calcilutite.
- ellowish-gray clayey, phosphatic, dolomitic quartz sand.
Olive-gray clayey, phosphatic quartz sand with chert fragments.
Olive-gray to light olive gray clayey, phosphatic quartz sand.
SGrayish orange to moderate yellowish-brown sandy, phosphatic clay.
Olive gray clayey, calcilutitic, phosphatic quartz sand.
Grayish-brown calcilutitic, clayey, phosphatic quartz sand.
Yellowish-gray calcilutitic, phosphatic quartz sand with limestone fragments.
-Dark greenish-gray calcilutitic, sandy clay.
Dark yellowish-brown clayey, phosphatic quartz sand with limestone fragments.
ery light-orange to light olive-gray sandy, calcilutitic, phosphatic clay.
Ten split-spoon cores were taken between 10 and 111 feet bls.
and display significantly different hydraulic conductivities.
to 32 and 43 to 45 feet bls intervals are quartz sandy dolosilts.
from 110 to 111 contained substantial quartz in the clay.
Pleistocene to Holocene undifferentiated quartz sands.
Figure 12: Columnar section for well #10 (W-16207).
- 20 Greenish-gray to light grayish green clay.
White to very light-grey clayey quartz sand.
Grayish-green to light grayish-gren clay.
S30 Yellowish-gray to graylsh-green sandy, phosphatlc clay with shark teeth, mollusks, and dolosilt.
Yellowlshgrey to grayish yellow clayey, dolomitic, phosphatic quartz sand.
Yellowlsh-grey sandy, clayey, phosphatic dolosllt.
Light grayish-green to graylih-graen phosphatlo, dolomltic clay.
-40 Yellowlsh-gray sandy, phosphatic doloillt.
S*- Very light-orange to yellowish gray dolomltic, phosphatic quartz sand.
-60 -Light olive-gray to moderate olive brown sandy, phosphetic clay with shark teeth.
-70 ht W tvery lightgy.ndy, phophatlc dolomite.
ht *yisFh-ngreen to ig rnlsn.ray phatic doloslt.
Light Oli-gray to light grenl. l wclay.
L Light olive-gray to grayish green phosphatic dolosilt.
SVary light-gray to white phosphatic, fossiliferous dolomite.
Yellowish.gry to very dark-purple sandy, phosphatic, fossiliferous dolosilt.
110 -- Moderate light-gray to moderate dark gray sandy, phosphatic clay.
120 Very light-gray to light gray sandy, phosphatic dolomite.
.White to very light gray clay.
-160 t White to very light gray sandy, phosphatic calcllutite.
Yellowish gray to light olive-gray dolomitic, phosphatic quartz sand.
Very light gray to yellowish gray undy, phosphatic dolomite.
-170 Very light-gray to yellowish-gray dolomitic clay with interbedded micrita.
I .I.. Yellowish-gray to white sandy, phosphatic calcllutlte.
White to light olve-gray fossilifrous limestone.
undoubtedly contribute to the lower permeability of this interval.
francolite are the most common minerals in the bulk samples.
locally between some of the parameters studied during this project.
are outlined in the following section.
of very fine to coarse, with modes of either fine or medium size.
undifferentiated intervals or Hawthorn Group.
samples are from intervals containing abundant quartz sand.
permeability of the undifferentiated Pleistocene-Holocene sands.
finer resolution in the permeameter sampling interval in each well.
Wiley and Sons, 463 p.
John Wiley and Sons, New York, p. 44-49.
Geological Survey Open File Report 21, 26 p.
Florida Geological Survey Bulletin 51, 164 p.
Appendix I: Lithologic logs for the 10 study cores.
._ -- .. .. .
19 SAMPLES FRONT 11 TO 90 FT.
OTHER FEATURES: UNWASHED SAMPLE; .
9 SAMPLES FROM 11 TO 53 FT.
INDEX FOSSILS INDICATIVE OF CRYSTAL RIVER FM.
19 SAMPLES FROM 11 TO 101 FT.
8 SAMPLES FROM 10 TO 42 FT.


text:
Home > History > Battle of Antietam (Sharpsburg)
Battle of Antietam (Sharpsburg)
This page tells all about the Battle of Antietam, the deadliest one-day battle in American history.
Battle of Antietam
The Liberators
The Battle of Antietam occurred on September 17, 1862, in Sharpsburg Maryland. The battle was named after a creek in the region. It was the bloodiest one-day battle in American history. As Robert E. Lee and his Army of Northern Virginia attempted to assert the will of the Confederacy in the North, they were followed by Union General George B. McClellan and the Army of the Potomac into Maryland. Lee, believing the majority of Marylanders were being held in the Union against their will, thought he and his army would be regarded as heroes or liberators as he entered the border state. In reality, however, the sentiments of western Marylanders had turned toward the Union, and Lee's Army was regarded with suspicion.
Serendipity Wasted
Before the battle began, Union forces had serendipitously found a copy of Lee's battle plan wrapped around three cigars. The plans indicated that Lee had divided his army and sent brigades to Harper's Ferry, Virginia, and Hagerstown, Maryland. This allowed for Union forces to attack each brigade in isolation, but General McClellan failed to act quickly enough and the opportunity was lost.
McClellan's Indecision is Costly
On September 15, General Robert E. Lee positioned his army of 18,000 in a defensive position behind Antietam Creek in the town of Sharpsburg. McClellan's Army arrived in town the same night with over four times the number of soldiers as Lee's Army. McClellan, however, overestimated the strength of the Confederates, causing him to delay an attack. During the delay, Lee was reinforced by the corps of General Longstreet and Jackson.
Control of the High Ground near the Dunker Church
The Dunker Church today at Antietam National Battlefield
On the morning of September 17th, General McClellan ordered General Joseph Hooker to attack the Confederate left flank for the purposes of occupying the high ground near the Dunker Church. A vicious and violent battle ensued in the cornfield adjacent to the church featuring bayonet charges, hand-to-hand combat, and gun battles at close range. Hundreds of soldiers were killed in this initial combat, and neither side had gained a clear advantage. One brigade, known as the Louisiana Tiger Brigade lost 323 out of 500 men when they were isolated. Hooker's brigade lost 2,500 men in a matter of two hours and historians believe the cornfield at the Antietam Battlefield changed hands at least fifteen times during the morning and afternoon. By the end of the morning, casualties for both sides numbered over 13,000.
Bloody Lane
Bloody Lane Today
In the afternoon, Union forces struck the center of the Confederate line in an attempt to divert some of their forces from a patch of forest known as the West Woods. Confederate forces were in a good defensive position atop a hill in an old, sunken road. From the Sunken Road, Confederate forces fired withering rounds into the Union brigades, inflicting terrible casualties. Union forces eventually exploited a weakness in the Confederate defenses and began to break through. Amidst growing confusion, the Confederates along the Sunken Road fled toward Sharpsburg. Union forces, however, were violently driven back during their pursuit by Longstreet's Brigade. All told, over 5,600 total casualties were recorded along the Sunken Road, which would eventually be called Bloody Lane.
Burnside's Bridge and the Confederate Retreat
Burnside's Bridge Overlooking Antietam Creek
Toward the later afternoon, the battle moved to the southern end of the battlefield, particularly for control of a bridge spanning Antietam Creek that would come to be known as Burnside's Bridge. Union Major General Ambrose Burnside ordered his soldiers to storm the bridge, where they took heavy fire from Confederate gunners. It took three separate attempts before Union soldiers successfully crossed. While Burnside's men crossed the narrow bridge with their artillery and wagons, General Robert E. Lee took the time to reinforce the right flank of his army. Numerous assaults and repulsions marked the remainder of the afternoon. By 5:30 p.m., it was clear there would be no winner. In total, the two sides suffered almost 23,000 casualties, making it the single bloodiest day in American military history. Although Lee expected another Union assault on the 18th, it never came, and an informal truce was established so each side could collect and tend to the injured and dead. On the night of the 18th, Confederate forces left Sharpsburg, crossing the Potomac River back to Virginia.
A Devastating Turn of Events for Robert E. Lee
Despite the fact that neither army was victorious, Union soldiers paraded through the streets of Frederick, Maryland, after the battle. The Confederate foray into Northern territory ultimately proved unsuccessful, delivering a crushing blow to its hope for foreign recognition. President Lincoln, meanwhile, saw Antietam as enough of a Northern victory to have the confidence to issue the Emancipation Proclamation five days after the battle. This landmark proclamation freed all slaves in "enemy territory" and ensured that slavery would cease to exist if the North were to win the war.
Antietam and Emancipation Proclamation Activities and Activities
The Importance of Foreign Recognition for the Confederacy
George McClellan Biography
Stonewall Jackson Biography
Robert E. Lee Biography
Battle of Antietam Paragraph Paramedics - Find and correct the spelling, punctuation, and usage errors in the paragraph. Click on the error and then type in the correction. Immediate feedback is given.
Emancipation Proclamation Reading Comprehension - Online - This resource includes a historical passage and ten multiple choice questions. It gives immediate feedback. In addition, when you click the "listen" button, you can hear the passage while it highlights the text.
Battle Breakdown: Antietam - This awesome printable helps students to essentially diagram battles! It requires students to write the cause, the players in the battle, the victor, and the effects of the battle. It comes with a descriptive narrative about the battle that students can use.
Battle of Antietam Fact or Fiction - Printable - This fun activity requires students to read a Battle of Antietam passage and then, to sort 10 statements into those that are facts and those that are fiction. The program gives immediate feedback.
Battle of Antietam Fact or Fiction - Online - This fun activity requires students to read an Antietam passage and then, to sort 10 statements into those that are facts and those that are fiction.
Battle of Antietam Correct-me Passage - Online - This fun activity requires students to correct a passage about the Battle of Antietam that has nine factual errors. Students first must discover the errors, then click on them and select the correct answer from the drop down menu.
The Great Squander: McClellan squanders an incredible opportunity at Antietam - This historical prompt challenges students to consider the great opportunities squandered by Union General George McClellan prior to Antietam. Then, to think of and write about a time when they may have squandered a "golden" opportunity.
The Emancipation Proclamation - I'll Always Remember Where I was... - This historical prompts requires students to imagine the earth-shaking effect the news of the Emancipation Proclamation had on Americans in 1862. In the spirit of the Emancipation Proclamation, students must write about the biggest news event of their lives and describe its impact.
The Great Squander at the Battle of Antietam
Battle of Bull Run II (Manassas)
Battle of Fredericksburg
Emancipation Proclamation Reading Comprehension - Online
James Ewell Brown "Jeb" Stuart Biography
James Longstreet Biography


text:
Massive Load-Shedding Turns Working Schedule Upside Down
Admire Mugodhi, who runs a small welding enterprise that manufactures door and window frames and an assortment of other metal products, says he has had to change his work routine to fit the load-shedding schedule.
"We only get electricity after 10 at night," he says as he sets up his equipment in preparation to start work.
"As you can see, all my equipment uses electricity, there is no way I can operate without electricity.
"Since we only get power at night that means I can only work at night."
He says he now spends his day in bed.
But this seemingly punishing routine has had its downside on his business as walk-in clients dwindle.
And also working at night, he adds, compromises the quality of his products.
David "Diva" Nyandima, who runs a small confectionery enterprise in the same high-density suburb, where he backs, cookies and scones, says he is slowly getting used to the new unfamiliar routine.
"I am yet to get used to working at night," he says.
"Sometimes I doze off and some of my products get burnt in the oven.
"Products that get burnt obviously have to be thrown away and this is costing me a lot of money."
What makes his job doubly taxing is that in the afternoon he has to sell his wares.
While some enterprising entrepreneurs have now resorted to using diesel-powered generators and small solar power units, this proposition is a non-starter for power-intensive businesses.
Even for those who use solar and diesel generators, the cost of acquiring them is prohibitive.
The cost of fuel, which is usually acquired from the black market, also remains forbidding.
For Nyararai Muteve – a welder a Highfield's famous and busy Gazaland Shopping Centre — he has had to switch to liquid petroleum gas (LPG) to power his welding machine.
But this comes at a price.
LPG as selling for anything between $13 to $15 per kilogramme, which is more or less his monthly electricity bill.
It puts an extra cost on the business.
The challenges the small businesses are facing are shared across the country.
Informal sector
This has a heavy toll on the informal sector, which is one of the biggest employers in the country.
In 2017, the International Monetary Fund (IMF) indicated that the country's informal sector could be the sixth largest in the sub-Saharan region, contributing between 40 percent and 50 percent to economic growth.
SME Association of Zimbabwe founding chairperson Mr Farai Mutambanengwe says most small businesses have taken a hit from the rolling power outages.
"Most small businesses have been forced to work during night-time hours, which does not augur well for business," says Mr Mutambanengwe.
"Obviously the situation does not bode well for business because if you have no power, then you do not work.
MORE: Sunday Mail
Tags: Electricity load shedding ZESA
Tuck-shops Under The Spotlight For Unscrupulous Business Practices
Ex- Wenela Members Facing Documentation Challenges
Cyclone: IOM appeals for US$7,2m
Heat Waves Killed 36 In india
State applies for Kasukuwere's property forfeiture


This probably goes without saying, but you should try to avoid bending your new iPad Pro. A new video by a popular YouTuber demonstrates why.
JerryRigEverything's Zach Nelson has conducted his usual routine of damage testing of the new 2018 iPad Pro. He scratches the tablet, burns the display, and subjects it to other forms of arguably unnecessary device-violence to test its durability.
The video culminated in Nelson's signature bend test. Unsurprisingly, the YouTuber was able to bend and destroy the new iPad Pro without much exertion or force.
But to call that a startling revelation is probably a stretch. The new iPad Pro is an extremely thin piece of metal, glass, and plastic. It's going to bend — and possibly be irreparably destroyed — if you intentionally put pressure on it.
While there are justifiable concerns with the new iPad Pro's durability, they aren't specific to the new Apple tablets. All tablets are prone to bends since they're larger and have more surface area than smartphones. That's especially true as tablets have gotten thinner over the years.
Because of that, users should take obvious precautions. If you carry the iPad Pro in a backpack or other bag, make sure that it's positioned in a way that won't put too much pressure on it. Similarly, you should avoid keeping it on a couch or seat where it might be sat on.
On the other hand, Nelson's conclusion that the new iPad Pro has a design flaw or hasn't undergone rigorous testing likely isn't entirely accurate. Again: it's a large, thin piece of metal and glass. It's going to break if bent.
In other words, the results of the video are pretty obvious. This video probably isn't going to spawn a new "Bendgate" controversy. And if it does, it'll undoubtedly be fairly unfounded.
Smartphones, being fairly rigid devices that are often carried in a pocket, should undoubtedly be fairly resistant to bending. But a tablet doesn't need to meet that requirement. As Nelson points out in the video, you aren't going to keep your new iPad Pro in your back pocket.
The iPad Pro is a thin but expensive device — and users should treat it as such. If you're particularly concerned about its durability, buy a case or keyboard cover and take care of it.