text:
\section{Introduction} \label{introduction}

The field of gamma-ray ($>$100 MeV) astronomy has long been plagued by
the double problem of low number of photons (and hence, sources), and
their poor  characterization. Indeed, the angular resolution of gamma-ray
instruments is typically measured in degrees or arcminutes at best, also being a strong
function of energy (improving with increasing energy).  Thus, one of
the earliest gamma-ray catalogs, the Second COS-B
catalog (2CG), contained only 25 sources and 21 of them were {\it
  unidentified}~\citep{COSB}, while two decades later, the Third EGRET Catalog (3EG)
contained 271 sources ~\citep{3EGcatalog}, almost two thirds of which remained {\it
  unidentified}, despite intense follow-up efforts~\citep{Thompson08}.

 The {\it Fermi} Large Area Telescope (LAT), launched in 2008, represents a giant leap in capabilities
  compared to past instruments. With its silicon-strip detector technology, wide
  field of view (2.4 sr), and high duty cycle ($>$95\%), the LAT has
  not only already detected over 1000 times more photons than EGRET did, but
  these cover a far broader energy range (20 MeV to $>$300 GeV)
  and are much better characterized~\citep{LATinstrument}.

Within a few months of launch, the LAT team released a first list of 205
highly-significant ($>10 \sigma$) sources. Known as the {\it Bright
  Source List} (also referred to
as 0FGL~\citep{FGL0}), this list represented a big improvement over
previous catalogs, as illustrated by the fact that fewer than 20\% of
0FGL sources were {\it unassociated}\footnote{The {\it identification} of a gamma-ray
  source typically requires a correlated timing
    signature at different wavelengths, whereas an {\it association}
    is less stringent, being based
    only on positional coincidence. The greatly reduced
    positional uncertainties of LAT sources, compared to previous
    experiments, has reduced the number of plausible associations per
    source, making it now more useful to talk in terms of {\it unassociated} sources,
  rather than {\it unidentified} ones.} at the time of
publication\footnote{At present, only $\sim$5\% of 0FGL sources remain unassociated.}. Full-fledged {\it
  Fermi}-LAT catalogs have been released periodically since then,
based on 11 months~\citep[1FGL, ][]{FGL1}, two
years~\citep[2FGL, ][]{FGL2}, and most recently, four years of
data~\citep[3FGL, ][]{FGL3}.  The number of known ($>$ 100 MeV) gamma-ray sources now stands at
over 3000, with approximately one third of these in the {\it
  unassociated} category~\citep{FGL3}. Figure~\ref{source_fraction}
shows the fraction of 3FGL sources in the three broad categories,
Active Galactic Nuclei (AGN), pulsars (PSR), and Unassociated sources as a function of the (4-year) significance of the source. The sharp drop in
the fraction of gamma-ray sources that are pulsars, with decreasing
significance (from $\sim20\%$ of $>20\sigma$ sources, to $\sim10\%$ of
$\sim10\sigma$ sources), in contrast to the fraction of AGN
(a relatively uniform $\sim60\%$) suggests that the discovery space
for new (relatively gamma-bright) pulsars remains significant.

In addition to uncovering large numbers of new gamma-ray sources, the
improved sensitivity of the LAT also enables a much better characterization
of these sources, facilitating their identification. Among the
earliest scientific results from the LAT, was the discovery of a large population of {\it radio-quiet} gamma-ray
pulsars~\citep{LATBSPs, sazparkinson10}, using a new blind-search
technique developed specifically for the very long time series
expected with the LAT~\citep{atwood06}. These findings confirmed
some predictions that many of the unidentified EGRET sources in the Galactic plane were
pulsars~\citep[e.g.][]{Yadigaroglu95}, favoring {\it outer gap} pulsar
models~\citep{Cheng86,Romani96,romani14}, where the gamma rays are generated in
the outer magnetosphere, as opposed to {\it polar cap} models, where
the emission comes from closer to the neutron star surface~\citep[e.g.][]{harding98}.

Perhaps even more surprising than the large number of young, {\it
  radio-quiet} pulsars discovered by the LAT was the detection
of a large population of gamma-ray {\it millisecond} pulsars
(MSPs)~\citep{LATMSPs}, something which, though not entirely
unforeseen~\citep[e.g.][]{Harding05}, has exceeded all expectations. Through
the joint efforts of the LAT team and radio astronomers working at the major radio observatories around the world,
the {\it Fermi} LAT Pulsar Search Consortium (PSC) has been carrying out
extensive radio observations of both newly-discovered LAT gamma-ray pulsars, as well
as carrying out new pulsar searches in LAT {\it unassociated}
sources~\citep{PSC}. This has led to the discovery of over 70 new
pulsars to date~\citep[e.g.][]{ransom11}, the vast majority of which
are MSPs\footnote{For the latest list of LAT-detected gamma-ray pulsars, see {\tt https://confluence.slac.stanford.edu/x/5Jl6Bg}}.

Aside from pulsars, there are many classes of astrophysical objects that emit gamma
rays. In fact, the 3FGL Catalog lists around twenty different gamma-ray
source classes. By far the two largest classes are,
broadly speaking, pulsars (PSR) and Active
Galactic Nuclei (AGN), especially those of the {\it blazar}
variety~\citep{FGL3}. Indeed, AGN detected by the LAT can be further
subdivided into many different classes (e.g. flat-spectrum radio quasars
(FSRQs), BL Lacs, etc.). For extensive details, including the latest catalog
of LAT-detected AGN, see \cite{3LAC}.

It has been known for some time now that the two main classes of
gamma-ray sources (AGN and PSR) can be roughly distinguished by their timing and spectral
properties: AGN display variability on month-long time scales and
have energy spectra that break more softly than pulsars in
  the LAT energy band. Pulsars tend to be {\it non-variable}
(on long time scales) and have spectra with more {\it curvature},
breaking on both ends, and therefore poorly described by a simple power law, normally
requiring the addition of an exponential cut-off at a few GeV. This was illustrated graphically in the 1FGL catalog
by means of a Variability-Curvature plot~\citep[See Figure
8 from][]{FGL1}, showing pulsars and AGN clustering in opposite
corners. Indeed, a number of bright {\it pulsar candidates} were identified this
way~\citep[e.g.][]{J2339,J1311}, and later discovered to be pulsars~\citep{pletsch12,ray14}.

The large increase in the number of gamma-ray sources detected with the LAT, as
well as the somewhat crude and arbitrary (as well as subjective) nature of
visual inspection techniques, make it desirable
to develop an automated scheme to classify {\it candidate}
sources, according to their {\it predicted} source class. In recent years there has been an explosion of interest in {\it data
  science}, and the application of statistical techniques to all
fields, including astronomy~\citep[for a nice overview of recent developments in machine
learning and data mining techniques applied to astronomy,
see][]{DataMining}. Various groups have started applying these
techniques to astronomical data. Recently, ~\cite{WISE} applied the Random
Forest algorithm to the classification of variable stars using the
Wide-field Infrared Survey Explorer (WISE) data, achieving efficiencies of up
to $\sim$85\%. In the gamma-ray regime, a number of groups have
worked on both unsupervised learning~\citep{Lee12} and supervised
learning~\citep{SYBIL} techniques. Indeed, the {\it Fermi} LAT
Collaboration applied two different machine-learning techniques to the automatic
classification of 1FGL sources: logistic regression and
classification trees. By combining the two methods, a success rate of $\sim$80\% was estimated for the correct
classification of the gamma-ray source class~\citep{STATSFGL1}. An
artificial neural network approach has also been implemented and
applied to the 2FGL catalog with some promising preliminary results~\citep{SalvettiThesis}.

We have explored a large number of statistical techniques for ranking
data~\citep[e.g.][]{alvoyu} and have applied some of the most commonly used
algorithms to the problem of classification of LAT  gamma-ray sources
in the most recent 3FGL Catalog~\citep{FGL3}. Our goal is not to
  firmly establish the class of each of the $\sim$1000
  unassociated gamma-ray sources in 3FGL; rather, it is to provide an
  objective measure that quantifies the likelihood of each source of
  belonging to one of the two major classes (pulsar or AGN), for the purpose of aiding in the necessary
  follow-up studies and searches (mostly in other wavelengths) that
  {\it can} conclusively determine the nature of each individual
  source. In pointing out sources that are unlikely to belong to
  either of the big source classes, our study also serves to highlight
  gamma-ray sources that might have a more {\it exotic} origin (e.g. dark matter
  annihilation). Finally, our results may be useful for population studies,
  or to estimate the number of new gamma-ray sources in each class
  that we might expect to identify in the future.

The structure of the paper is as follows. In Section~\ref{dataselection} we describe the data sets used in
the paper as well as some of the key predictor parameters
employed. Section~\ref{algorithms} discusses the various algorithms we
considered, with an emphasis on those that proved most
successful (Random Forest and Logistic Regression).  Next, in
Section~\ref{application} , we discuss
the application of these algorithms to the 3FGL catalog and provide an
overview of our results. Finally, Section~\ref{discussion} provides a
discussion of our key results and conclusions, including our
predictions on the nature of both the {\it unassociated} sources, as well as certain
gamma-ray sources with known associations (e.g. gamma-ray binaries and
SNR/PWNe). We also provide a list of plausible X-ray counterparts for
some of our best {\it pulsar candidates}, obtained through our
follow-up program of LAT gamma-ray sources with {\it Swift}, {\it
  Chandra}, and {\it XMM}.

\section{Data and Feature Selection} \label{dataselection}

The {\it Fermi} LAT Third Source Catalog (3FGL) was publicly released
through the Fermi Science Suppport Center (FSSC\footnote{{\tt http://fermi.gsfc.nasa.gov/ssc/}}) in January
2015, with a few minor updates being posted around the time of official publication~\citep{FGL3}. The results presented here make use
of the uptdated version released on 2015 May 18 (FITS file gll\_psc\_v16.fit\footnote{The 3FGL catalog, along with other LAT catalogs, are also available as an R package, {\tt fermicatsR},
  from the Comprehensive R Archive Network webpage {\tt
    http://cran.r-project.org/web/packages/fermicatsR/}}). The 3FGL
catalog contains 3,034\footnote{Note that the various components
  associated with the Crab nebula are counted as different sources.} gamma-ray sources, of which 1010 are {\it unassociated}.

\subsection{Training and Testing Sets \label{sampling}}

Since we are interested in applying a number of {\it
  supervised} learning techniques to classify sources according to
their likelihood of falling into two broad classes (PSR and AGN), our first step involves selecting sources from
3FGL that are known to fall into these two categories.  We select all
sources that are identified or associated with pulsars (i.e. sources
classified as PSR or psr, respectively, in 3FGL), and all sources that are identified or associated
with any type of AGN, including FSRQs, BL Lacs, etc  (i.e. sources of the
following class: FSRQ, fsrq, BLL, bll, BCU, bcu, RDG, rdg, NLSY1, nlsy1, agn, ssrq, and sey\footnote{For a definition of all
these acronyms, see Table 6 of \cite{FGL3}.}). After filtering
out sources with missing values (six AGN and one
pulsar, namely PSR J1513--5908), we are left with a total of 1904
sources, of which 1738 are AGN and 166 are PSR. We created training and testing sets by randomly
selecting 70\% and 30\% of these sources, respectively. In summary, our
training sample contains 1,217 AGN and 116 PSR, while our testing sample
contains 521 AGN and 50 PSR.

Because we are also interested in the classification of pulsars into
`young' (YNG) and millisecond (MSP), we further split the known
gamma-ray pulsars into these two sub-samples. We make use of the public list of
LAT-detected gamma-ray pulsars\footnote{{\tt
    https://confluence.slac.stanford.edu/display/GLAMCOG/Public+List+of+LAT-Detected+Gamma-Ray+Pulsars}}
and cross-correlate these with the 3FGL catalog, obtaining a list of 142
known gamma-ray pulsars (77 YNG and 65 MSP) which, we randomly split up
into a training set containing 70\% of the sample (52 YNG and 47 MSP) and a
testing set with the remaining 30\% (25 YNG and 18 MSP).

\subsection{Feature Selection}

The 3FGL Catalog contains a large number of measured parameters on each source,
covering everything from the source positions and uncertainties, to
fluxes in various bands, etc. In addition to all the parameters
included in 3FGL,  we also defined the following {\it hardness
  ratios}, following \cite{STATSFGL1},  as:

$hr_{ij} = (EnergyFlux_j - EnergyFlux_i)/(EnergyFlux_j + EnergyFlux_i)$

where $i$ and $j$ are indices corresponding to the five different LAT energy
bands defined in the 3FGL catalog: i=1: 100--300 MeV, i=2: 300 MeV --
1 GeV, i=3: 1--3 GeV, i=4: 3--10 GeV, and i=5: 10--100 GeV, respectively. The energy flux in each band is computed
by integrating the photon flux, using the measured spectral index.

We started out with a large number (35) of potential parameters on which
to train our sample. Using a two sample t-test, we then determined how the
various parameters were correlated to each other (e.g. Signif\_Avg is
highly correlated with Flux\_Density) and removed highly correlated
($\mid$$\rho$$\mid >$0.7) parameters as well as
parameters related to the position of the source (e.g. GLAT, GLON). We then applied a log
transformation to some parameters that displayed highly skewed
distributions (e.g. Variability\_Index). We dropped sources with missing values for any of our predictor parameters, leaving
a total of 3021 sources, out of the original 3034. Table~\ref{variables} shows the 9 predictor parameters
that we ended up using in our various models for classifying PSR vs AGN, along with the range and
median of these parameters in our data sets. These include the
well-known {\it curvature} and {\it variability} parameters
(Signif\_Curve and Variability\_Index, respectively), the spectral
index and flux density of the source, the uncertainty in the energy
flux above 100 MeV (Unc\_Energy\_Flux100), and the four hardness
ratios constructed using the five different energy bands and equation described
above (i.e. $hr_{12}$, $hr_{23}$, $hr_{34}$, and $hr_{45}$). For the
MSP vs YNG classification, we added GLAT (Galactic Latitude) as a
possible predictor parameter. In Appendix~\ref{getdata} we provide the R
script used to obtain and {\it clean} the data, as well as to perform
our detailed feature selection.

\section{Classification Algorithms} \label{algorithms}

We considered a long list of algorithms (see Tables \ref{AGNvPSR} and
\ref{YNGvMSP} for a complete list), including Decision Trees,
Support Vector Machines (SVM), a simple logistic regression (LR) model (with both
forward and backward stepwise elimination), various modified versions
of LR (e.g. Boosted LR, logistic decision trees), Random Forest (RF), as well as some combination of methods (e.g. a 2-step method
involving decision trees followed by LR). We used the {\tt RWeka} package~\citep{RWeka,Weka}, along with the
{\tt pROC} R package~\citep{pROC} to draw Receiver Operating
Characteristic (ROC) curves, commonly used in the data mining
and machine learning communities to measure the performance of
a classifier. In such a curve, one plots sensitivity (true positive rate) vs
specificity (true negative rate) for varying thresholds, allowing us
to evaluate the tradeoffs involved in each choice, with the ultimate goal being to classify
correctly the largest proportion of pulsars in our sample, while keeping the proportion of
mis-classifications as low as possible. Figures~\ref{lr_roc} and
\ref{rf_roc} show the ROC curves for our two best algorithms in the
AGN vs PSR classification, while Figures~\ref{blr_pulsars_roc} and
\ref{rf_pulsars_roc} show the corresponding ROC curves for the best
two algorithms in the YNG vs MSP classification. We define the best threshold
as that which maximizes the sum of both terms (sensitivity + specificity, see Appendix~\ref{roc} for the relevant R
scripts). We also used the\texttt{ randomForest }\citep{RFR} R package to fit random
forests and the \texttt{e1071}~\citep{e1071} R package to fit
SVM models. Ultimately, we settled on the RF algorithm, for its overall
accuracy, and LR for its slightly better sensitivity to pulsar
classification. In the following sections we provide some details on the most successful models we decided to apply to the
3FGL Catalog.

\subsection{Random Forest}\label{RFsection}

Random forest~\citep[][hereafter RF]{RF} is an ensemble learning method which uses decision
trees as building blocks for classification, regression and other
tasks. By aggregating the predictions based on a large number of
decision trees, RF generally improves the overall predictive
performance while reducing the natural tendency of standard decision trees to
over-fit the training set. RF is essentially an extension of a Bootstrap aggregating method called
Tree bagging~\citep{breiman1996bagging}. First of all, tree bagging for classification
generates $B$ different training sets by sampling with replacement
from the original training set. Then it builds a separate tree for
each training set, resulting in $B$ fitted trees, and finally for each
new observation, it generates the predicted class probability by
taking the average of the $B$ predicted class probabilities from all $B$ fitted trees.

RF provide an improvement over bagged trees by way of a
random small tweak that decorrelates the trees. Its main idea is that
in each step of identifying the best split of a node in the tree growing stage, a
random sample of $m$ parameters drawn from all the predictor parameters
is used for consideration in selecting the best split for that node.
Typically, a value of $m\sim\sqrt{p}$ is used, where $p$ is the total number of
parameters tried at each split. By forcing each split to consider only a subset of the predictor
parameters, the splits will not always be constructed from the
strongest predictor but some other potential strong predictors,
thereby making the RF more reliable.  It is thus advantageous to use RF when there exists multicollinearity
in the predictor parameters. The detailed algorithm can be found in
the book by \citet{james2013introduction}.

In this paper, we make use of the \texttt{randomForest} package \citep{RFR} in R. In order to ensure the stability of our
results, we grew a large number of trees (10,000 vs the default of
500) and scanned over a range of values of $m$. We found that a value of
$m$=2 was optimum, giving an OOB\footnote{``Out-of-bag'', in the sense
  that it is based on the portion of the data not already used for training the
  original tree, thus providing an internal estimate of the error
  which is expected to be comparable to that obtained with a final
  independent {\it testing} data set. See \cite{RF} for details.} estimate of the error of 2.3\%.

The \texttt{randomForest} package can also generate a variable importance measure for each parameter in terms of the mean decrease
in accuracy (MDA), which is computed from permuting the OOB data: For each tree,
the prediction error on the OOB portion of the data is recorded
(error rate in the classification). Then the same is done after permuting each predictor variable. The difference between
the two are then averaged over all trees~\citep{RFR}.

The \texttt{randomForest} package
also computes the {\it proximity} measure, which, for each pair of
elements (i, j), represents the fraction of trees in which elements i
and j fall in the same terminal node. This can be used to calculate
the ``outlyingness'' of a source, as the reciprocal of the sum of
squared proximities between that source and all other sources in the
same class, normalized by subtracting the median and dividing by the
median absolute deviation, within each
class~\citep{RFR}. Figure~\ref{outlyingness} shows the {\it outlyingness} of all 3FGL
sources, with respect to the PSR and AGN classes. Note that most
pulsars have large values of ``AGN outlyingness'', while most AGN have
large values of ``PSR outlyingness''. A large value of ``outlyingness'' along both
  axes could imply a different gamma-ray source class altogether
  (i.e. non-Pulsar {\it and} non-AGN).

\subsection{Logistic Regression}\label{LRsection}

Logistic regression (hereafter LR) is a very popular probability model that was developed
by \citet{cox1958} and \citet{walker1967}. The model can be
used to predict a binary response based on one or more predictor parameters
(features). For an 0/1 binary response variable $y$ and $p$ predictor
parameters $x_{1},...,x_{p}$, the logistic regression model can be
written as
\[
y\mid x\sim Bernoulli(\pi)
\]
and
\[
logit(\pi) \equiv \log\frac{\pi}{1-\pi}=\beta_{0}+\beta_{1}x_{1}+...+\beta_{p}x_{p}
\]
where $\beta=(\beta_{0},\beta_{1},...,\beta_{p})$ is a vector of
unknown parameters which can be estimated by the maximum likelihood
method. For further details on logistic regression, including its use
as a classifier, see \cite{LR}.

Both forward and backward stepwise methods are considered in parameter
selection. Forward stepwise method starts with no predictor parameters
in the model, and then recursively adds parameters one at a time
according to the Akaike Information Criterion (AIC).
Backward stepwise method is performed similarly except that it starts
with all predictor parameters in the model and proceeds by dropping
parameters one at a time. In our study, we found that both methods gave the same result.

\subsection{Boosted Logistic Regression}\label{BLRsection}

Boosted logistic regression is a powerful classifier based on additive
logistic regression fitted by stage-wise optimization of the Bernoulli
log-likelihood \citep{friedman2000additive}. In a two-class problem,
the additive logistic regression is of the form:
\[
\log\frac{P(y=1\mid x)}{P(y=0\mid x)}=\sum_{m=1}^{M}f_{m}(x)
\]
where $f_{m}(x)$ is learned in fitting the $m$th logistic regression
on a weighted training data. Its basic idea is to adaptively change
the frequency weights of the observations in the training data according
to the performance of the previous logistic model. Unlike fitting
a single model to the data which may not fit the data well or potentially
suffer from over-fitting, the boosting approach instead, by sequentially
fitting the logistic regression models, gradually improves the model
fit in the regions where it originally did not perform well.

\citet{friedman2000additive} developed an algorithm for fitting additive
logistic regression models, named as the LogitBoost algorithm which
can be implemented using the LogitBoost function in the RWeka package
\citep{RWeka,Weka}.

\subsection{Model Building Procedure}

In order to determine the optimal cutoff probability value of the
prediction generated from each model, we adopted a 10-fold
cross-validation method. The procedure of model building is summarized as follows:

\begin{enumerate}
\item We partitioned our 3FGL data set at random into 70\% for
  training and 30\% for testing, as described in Section~\ref{sampling}.
\item We divided the training set randomly into 10 equal-size subsets.
\item We used 9 subsets to build a model and apply the fitted model to
  test on the remaining subset. We then repeated this procedure for
  all 10 subsets until all the subsets were tested.
\item We obtain the ROC curve based on all the tested subsets and determine the best cutoff value.
\item Using the training data to build a model, we then apply the fitted model and best cutoff value to generate our prediction (``PSR" or ``AGN") for the testing data.
\end{enumerate}

\section{Application of the algorithms to 3FGL} \label{application}

Figure~\ref{mda1} shows the relative importance of the input
parameters in the RF model, expressed as the mean
decrease in accuracy, as described in Section~\ref{RFsection}. Note that in some cases we
have transformed the parameters by taking the log, due to the skewness
of the distributions. We note that, perhaps not surprisingly, the
curvature significance and variability parameters (Signif\_Curve and
Variability\_Index) are two of the three most important predictor
variables, while the uncertainty in the energy flux
(Unc\_Energy\_Flux100) also turns out to
be very important, perhaps as a proxy for the quality of the spectral
fit of the source. In the case of the classification of pulsars
into YNG and MSP, we added GLAT (Galactic Latitude),
which we found to be useful in discriminating between the two classes
(In fact, it is the second most important parameter, see
Figure~\ref{mda2}). Table~\ref{logit}, on the other hand, shows the values of the
parameters ($\beta_{0}, \beta_{1},...$) corresponding to the
various predictor parameters in the best logistic regression model
(backwards stepwise), giving also an indication of the significance of
each one (Variability\_Index being the most significant, in this case).

\subsection{Results}

After applying a large number of algorithms to the problem of
gamma-ray source classification, we concluded that the RF technique
provides the overall best accuracy\footnote{Defined as the
  number of correctly classified AGN and PSR, divided by the total
  testing sample size.} (96.7\%), while LR (with backward stepwise elimination)
provides only slightly lower overall accuracy (94.7\%), but a better
sensitivity to pulsar identifications (98\%, vs 96\% for RF). Table~\ref{AGNvPSR} provides a summary the results of all the
various algorithms we tried, as applied to the problem of classifying AGN and
PSR.

The performance of the models, given in Tables~\ref{AGNvPSR} and
  \ref{YNGvMSP}, is clustered into two groups: the best algorithms,
which perform basically the same, and the worst algorithms, which are
also basically the same. The best algorithms are all those tree-type
models such as Decision Tree and RF (CV) while the worst ones are
those linear-type models such as LR and Boosted LR (CV). Therefore,
tree-type models seem better than linear-type models in classifying
AGN and PSR. From Table~\ref{YNGvMSP}, the best performing algorithms
in the case of YNG vs MSP are Boosted LR and RF, which are the only two
ensemble methods considered here.  Ensemble methods aim at combining
many models to form the final classification (e.g., RF combines many
tree models while boosted LR combines many LR models). Therefore, we
believe that combining models can help improve the classification of
young vs millisecond pulsars. The small scatter in the performance of
the algorithms is likely due to the imbalanced nature of the data
sets. Indeed, we see that even the worst performing algorithm has an overall accuracy
of $>$93\%. It is worth considering an additional performance measure
called F1 score (or F-measure), which is defined as the harmonic mean of precision
and recall, therefore conveying a balance between these two
quantities~\citep{F1}. This F1 score for PSR ranges from 0 to 1, with
a larger value implying a better performance in classifying PSR. The
F1 scores have a wider range (see last column in Table~\ref{AGNvPSR}),
and the four tree-type models perform better than the linear-type models for the classification
of PSR and AGN, with the RF still returning the best performance.

To test the robustness of our results, we ran both the RF and LR
algorithms ten times, randomly selecting different training and
testing sets in each case and found consistent results. Both
methods returned an overall accuracy of $\sim$96\%, with the RF
technique performing marginally better than LR.

For our analysis of the pulsar population, using a much smaller sample of 142 known
gamma-ray pulsars (77 YNG and 65 MSP), we again found that the RF
algorithm returns the best overall accuracy (90.7\%), while a boosted
logistic regression analysis comes a close second (88.4\%). We
caution, however, that these results are based on a relatively small testing
sample of only 43 pulsars (25 YNG and 18 MSP). Table~\ref{YNGvMSP}
summarizes the results obtained by the various algorithms in the
classification of young (YNG) vs millisecond pulsars (MSP).

Having settled on the best models, we then applied these to the
entire 3FGL catalog (that is, all 3,021 sources for which predictor
parameters are available and for which our models could therefore be
  applied). Table~\ref{allresults} shows a portion of these results
  (the full table being available electronically from the
  journal\footnote{Also at {\tt
      http://www.physics.hku.hk/$\sim$pablo/pulsarness/Step\_08\_Results.html}}
  \footnote{Also at {\tt
      http://scipp.ucsc.edu/$\sim$pablo/pulsarness/Step\_08\_Results.html}}). In
  the next section we go over some of the implications and follow-up
multi-wavelength studies based on these results.

\section{Discussion and Conclusions} \label{discussion}

One of the main goals of our investigations is to identify the most
promising {\it unassociated} gamma-ray sources to target in pulsar
searches, both in blind gamma-ray searches and in radio
searches. As discussed in Section~\ref{introduction}, these searches have been very
fruitful in the past, and as indicated by Figure~\ref{source_fraction}, the discovery potential remains
significant.

Overall, we find that of the 1008 {\it unassociated} sources for which we have a
prediction, in 893 cases the RF and LR algorithms are in agreement
(334 being classified as likely PSR and 559 as likely AGN). Out of the 334 unassociated sources
classified as likely PSR, 309 resulted in a consistent sub-classification using
both the RF and Boosted LR algorithms (194 of these being classified
as likely YNG pulsars, while 115 as likely MSPs). In Table~\ref{unassoc} we provide a list of
the most significant ($>10\sigma$) 3FGL unassociated sources which our methods (both
RF and LR) predict to be pulsars. While a $10\sigma$ cutoff is
somewhat arbitrary (and since we provide predictions for all sources,
searchers are free to set their own thresholds), we should keep in mind that no pulsars have been
found in blind searches of gamma-ray data below this significance (see
Figure~\ref{source_fraction}), so it is probably safe to say that most
pulsars found in the future will also be above this cutoff. Indeed, if
we consider only sources above 11$\sigma$ (roughly the lowest
significance for a pulsar found in a gamma-ray blind search), we note
that there are $\sim$1000 sources, of which only $\sim$125 are unassociated. Our
algorithms predict that roughly 75\% of these should be pulsars, of
which two thirds are predicted to be YNG and the remaining third
MSP. As an indication of how realistic these numbers are, we
point out that the discovery of an additional $\sim$90-95 LAT pulsars
would bring the percentage of pulsars within these $>11\sigma$ sources
up to $\sim$22\%, or roughly the same percentage found among the 239
most significant ($>32.9\sigma$) 3FGL sources, 100\% of which have
known associations. Of course, the well known lack of correlation
between radio and gamma-ray fluxes of pulsars also means that radio searches of less significant LAT
sources (led by the PSC) will continue to produce new gamma-ray pulsar
discoveries. It is likely that these will mostly continue to be in the
MSP category, since a significant fraction ($\sim$50\%) of the young
gamma-ray pulsars that are below threshold for LAT blind search discoveries will likely turn out to be radio-quiet, while
a large fraction of young pulsars that lie along the Galactic plane have probably
already been discovered in existing deep radio surveys.

We note that some of our predictions have already been confirmed by
the latest pulsar searches. Recently, for example, the young
pulsar J1906+0722 was discovered in a gamma-ray blind search of
3FGL J1906.6+0720~\citep{clark15} while the millisecond
pulsar (MSP) J1946-5403 was discovered in a targeted radio search of
3FGL J1946.4-5403~\citep{camilo15}. In some cases, athough pulsations
have not yet been discovered, the presence of a millisecond pulsar is strongly
suggested by multiwavelength observations
(e.g. 3FGL~J1653.6-0158~\citep{romani14}, 3FGL~J0523.3-2528~\citep{strader14,xing14},
3FGL~J1544.6-1125~\citep{bogdanov15},
3FGL~J2039.6-5618~\citep{salvetti15,romani15}), in
agreement with our predictions (see Table~\ref{unassoc}).

Searches for new gamma-ray pulsars need not be limited to {\it
  unassociated} sources. Indeed, other known gamma-ray source
classes like supernova remnants (SNRs) and pulsar wind nebulae (PWNe)
are {\it known} to be related to pulsars, and it is often hard to
disentangle the emission coming from the pulsar from that of the
remnant or PWN. Thus, it is worth looking more closely into those
sources that have been classified as SNR/PWNe, in the hope that a new
(as yet undiscovered) pulsar could be found in its
midst. Table~\ref{snrpwn} provides our model (LR and RF) predictions
for 3FGL sources with claimed SNR or PWN associations\footnote{We
  include 3FGLJ1119.1-6127  and 3FGLJ1124.5-5915, even though these sources are formally associated with
  PSRs J1119-6127 and J1124-5916 in the 3FGL catalog, rather than with
  their respective SNRs.}, which includes most of the likely GeV SNRs
(27/30) and about half of the marginal ones (8/14), as reported
in the ``First Fermi LAT Supernova Remnant
Catalog''~\citep{SNRCat}. Of the 9 SNRs from the LAT SNR catalog not
listed here (3 firm and 6 marginal), one corresponds to a LAT source
dropped from our analysis, as described in Section~\ref{dataselection}
(3FGLJ2021.0+4031e, associated with Gamma Cygni), while the remaining
8 have no corresponding 3FGL source. We find that a significant number of these sources are classified by both the RF and
LR algorithms as likely pulsars, in some cases with very large
probability. Indeed, as many as 14 of the likely SNRs from the LAT SNR
catalog have a P$>$0.95 in the LR algorithm, scoring highly in the RF
algorithm too, including such famous SNRs as IC443, Cas A, or the
Cygnus Loop. For more details on other potential associations we recommend
consulting the ``Census of high-energy observations of Galactic
supernova remnants\footnote{{\tt http://www.physics.umanitoba.ca/snr/SNRcat/}}''\citep{Ferrand12}.
We should add that \cite{SNRCat} reported an upper limit
of 22\% on the number of GeV candidates falsely identified as SNRs, so finding 6--7 new
gamma-ray pulsars among these sources would still be consistent with
the LAT SNR Catalog results.

Gamma-ray binaries are another class of gamma-ray source that has been
predicted to be associated with pulsars~\citep{Dubus06,Dubus13}. Looking at the results of our models
as applied to the four LAT-detected gamma-ray binaries (Table~\ref{binaries}), we see that
all of them are, indeed, predicted to be pulsars (specifically, of the
YNG variety). It may, therefore, be worth considering our YNG {\it
  pulsar candidates} as also being {\it gamma-ray binary candidates},
especially as it may be simpler to discover a gamma-ray binary orbital
modulation than to discover pulsed emission from such systems.

We also considered sources with large values of {\it outlyingness},
as discussed in Section~\ref{RFsection}. Table~\ref{out} lists those
3FGL sources with large ($>$75) values of PSR and AGN
outlyingness. We looked into the five sources highlighted by \citet{SYBIL} as being
the top``outliers'', among the high-latitude 2FGL sources. Out of the
five, four are classified by our algorithms clearly as MSPs (indeed,
one of them, PSR~J0533+6759 has already been discovered), while the
remaining source (now known as 3FGL J1709.5--0335) is classified by
both our RF and LR algorithms as an AGN.

Finally, looking at the results of our predictions as applied to the set of 1904 3FGL sources associated with AGN
or PSR (i.e. our combined training and testing set), it is worth
considering how {\it consistent} the two algorithms are with each
other, in addition to how accurate they are. We find that RF and LR are in agreement in $>$95\% of cases (1825 sources). Of all of these,
we find only 13 sources where our predicted class differs from that
given in the 3FGL catalog. While it is perfectly natural to expect all
of these associations to be correct, given the small number, we
provide a list of these sources in Table~\ref{misclassifications}, in
case some may deserve further investigation. We note that in all but one case, this
misclassification involves a source that in 3FGL has been associated
with an AGN, while our algorithms predict a PSR (in most cases
of the MSP variety). In the case of PSR J1137+7528 (clearly identified in the LAT by its pulsations),
the reason for the bad model prediction is likely due to the poor
  spectral fit arising from the low source significance (4.3$\sigma$). Indeed, two out of the 5 spectral
energy bins in 3FGL are only upper limits, and the resulting power-law
fit provides a perfectly acceptable model for the spectrum, as is
usually the case with AGN.

\subsection{X-ray observations}

As discussed in Section~\ref{introduction}, our goal in applying
machine learning techniques to the entire 3FGL catalog was not so much
to establish conclusively the class of individual sources, but rather
to identify the most promising sources for further investigation.

Uncovering the nature of gamma-ray sources usually requires a coordinated
multi-wavelength effort with many instruments. X-ray observations can be particularly
useful in blind searches for gamma-ray pulsars, given the much better
angular resolution of X-ray instruments and the sensitivity of pulsar
observations to uncertainties in the position being searched~\citep{dormody11,sazparkinson14}. Furthermore, X-ray
observations of pulsars are also beginning to shed light on possible
differences between {\it radio-loud} and {\it radio-quiet}
pulsars~\citep{Marelli15}.

In this section we describe our efforts to use X-ray observations
in the search for new gamma-ray pulsars among some of the most
promising unassociated LAT sources. Over the past several years, we
have observed a number of bright LAT unassociated gamma-ray sources with {\it Chandra} and {\it XMM}, currently the most sensitive
instruments in the $\sim$1--10 keV band~\citep[e.g.][]{sazparkinson14}. It is beyond the scope of this paper to
carry out an exhaustive analysis of all the X-ray observations of LAT
sources, many of which have, in any case, been published and discussed
elsewhere~\citep[e.g.][]{cheung12}. Here, we briefly discuss six interesting LAT sources for
which we (PI: Saz Parkinson) obtained either {\it Chandra} or {\it
  XMM} observations.

We performed a standard reprocess, analysis, and source detection in
the 0.3-10 keV energy band of the {\it XMM-Newton} and
{\it Chandra} observations, following~\cite{Marelli15}. For each of
the X-ray sources inside the gamma-ray error ellipse, we
performed a spectral analysis. After extracting the spectra, response
matrices, and effective area files, we fitted a power-law model using
either the $\chi^2$ statistic of the C-statistic~\citep{Cash79} in
case of a negligible background (the case of {\it Chandra} sources).
Unfortunately, the low statistics in some cases prevented us from an accurate
spectral characterization. For sources with a low number of counts
(typically fewer than $\sim$ 30), we fixed the column density to the
value of the integrated Galactic N$_H$~\citep{Kalberla05} and, if
necessary, the photon index to 2. We computed the gamma-ray to
X-ray flux ratio. As reported in \cite{Marelli11,Marelli15}, this
could give important information on the nature of the source. Finally,
we computed the predicted 5$\sigma$ upper limit on a detection, based
on the signal-to-noise. The detailed results of our X-ray analyses are
presented in Table~\ref{counterparts2}. In the following paragraph we
summarize some of our key findings.

3FGL J1035.7-6720 and 3FGL J1744.1-7619 were both observed for $\sim$25 ks
with XMM (obsIds 0692830201 and 0692830101), and show the
presence of possible X-ray counterparts at (RA, DEC) of (158.8652,
-67.3371) and (266.0030, -76.3205), respectively. LAT sources
3FGL J0212.1+5320 and 3FGL J0933.9-6232, both strong MSP candidates, were
observed with {\it Chandra} for $\sim$30 ks (obsid 14814) and $\sim$45 ks (obsid 14813), respectively, and show
the nearest possible X-ray counterparts at (33.0439, 53.3607) and (143.5049,
-62.5077), within the LAT error ellipse (e.g. see Figure~\ref{J0212}). Source 3FGL J1214.0-6236, coincident with
SNR G298.6+0.0 was observed with {\it Chandra} for $\sim$20 ks (obsid
14889) and shows a potential counterpart at (183.4307, -62.5936), as shown in Figure~\ref{J1214}. 3FGL J1405.4-6119, coincident with
SNR G311.5+0.3, was observed with {\it Chandra} for 13 ks (obsid
14888), showing a possible counterpart at (211.3103, -61.3077).

In addition to the dedicated {\it Chandra} and {\it XMM} observations,
many other LAT sources have been observed with less sensitive
instruments. Indeed, since the launch of {\it Fermi}, we have been
using the {\it Swift} X-ray telescope~\citep{gehrels04} to carry out follow-up observations of LAT unassociated
sources (\cite{stroh13}, Falcone et al.  (2016), {\it in preparation}), as part of an ongoing multi-year {\it
  Fermi} Guest Investigator program (PI: Falcone). This has resulted,
to date (through 2015-12-21), in the detection of almost 1900 sources
(with SNR $>$ 3)\footnote{{\tt http://www.swift.psu.edu/unassociated/}}. We selected only those X-ray sources within a radius of 1.2 times the
semimajor radius of the 95\% confidence region of our best pulsar
candidates (those predicted to be pulsars by both the RF and LR
methods). Table~\ref{counterparts} provides a list of these $\sim$90
potential counterparts. For each potential counterpart we convert the
measured count rate into an estimated flux in the 0.1--2.4 keV band by assuming a
power law spectrum of index -2 and estimating the Galactic Hydrogen
column density using the method of \cite{Willingale13}. Given the known
faintness of pulsars in the X-ray band~\citep[e.g.][]{Marelli11}, our flux estimates can be used
to likely rule out any bright {\it Swift} source (e.g. with flux greater than
$\sim5\times10^{-13}$ erg cm$^{-2}$ s$^{-1}$) as a plausible X-ray
counterpart of a pulsar \footnote{On this point, see also presentation
  by A. Falcone at the Sixth International Fermi Symposium, {\tt http://fermi.gsfc.nasa.gov/science/mtgs/symposia/2015/}.}
While in some cases the {\it Swift} observations have been superseded by observations with more sensitive instruments (e.g. 3FGL J0212.1+5320,
see previous paragraphs), we nevertheless leave the {\it Swift}
results in Table~\ref{counterparts} for completeness, and possible comparison.

redpajama_set_name: RedPajamaArXiv


1) Surveyors around the world are struggling with many current challenges. However, this article gives you a possibility to reflect about the impact of surveyors through history in the development and mapping of societies. This paper is a historical outline of the early land surveyors importance to the history of New Zealand, as they were among the advance guard of European settlers to walk the land and assess its potential for future development. Surveyors around the world are struggling with many current challenges. However, this article gives you a possibility to reflect about the impact of surveyors through history in the development and mapping of societies. The paper is a historical outline of the early land surveyors importance to the history of New Zealand, as they were among the advance guard of European settlers to walk the land and assess its potential for future development. We are pleased to share this paper with you since FIG Institution for the History of Surveying and Measurement organises a very special trip, conference and event on Charting and Mapping the Pacific Paradise of the Pitcairners at Norfolk Island, (an island half way between Australia and New Zealand), 6-10 July 2014: Invitation and program.
A version of this paper was presented to Celebrating the Past - Redefining the Future, New Zealand Institute of Surveyors Conference, University of Otago, Dunedin, New Zealand, 27-30 August 2013. In addition, much of this essay is drawn from Giselle Byrnes, Boundary Markers: Land Surveying and the Colonisation of New Zealand, Bridget Williams Books, Wellington, 2001; see also G. Byrnes, 'Surveying—Maori and the Land: An Essay in Historical Representation', New Zealand Journal of History, vol. 31, no. 1 (1997), pp. 85-98. See also Giselle Byrnes, 'Boundary Makers: Land Surveying in 19th Century New Zealand' in Mick Strack, ed., Survey Marks: A 2013 Celebration, School of Surveying and the New Zealand Institute of Surveyor, Dunedin, 2013, pp. 7-16.
Land surveyors in nineteenth-century colonial New Zealand were located, quite literally, at the 'cutting edge' of the great British imperial project to claim and tame new territories. The early land surveyors were important actors in this country's history as they were among the advance guard of European settlers to walk the land and assess its potential for future development. Indeed, the landscape of modern New Zealand testifies to their work through the place-names they assigned and which are still visible on the map. Moreover, most colonial land surveyors were aware that they were not 'first-time explorers', but were traversing landscapes that were already known, named and mapped by the indigenous Maori. In the early period of organised British settlement, from the 1840s to the 1860s, the land surveyors' efforts to claim the land were, therefore, much less an exercise in possessing the land outright than they were translating the meaning of land from one cultural framework into another. From the 1860s, however, and especially with the aggressive activities of the Native Land Court from 1865, the work of the colonial land surveyors took on a more potent role as Maori land was permanently transitioned from customary tenure to Crown-derived titles and subsequent private ownership. This essay briefly considers the colonising efforts of the early colonial land surveyors in New Zealand during the second half of the nineteenth-century following the assertion of British sovereignty 1840 and their negotiation of both the cultural and physical boundaries they encountered during the conduct of laying out the land for future settlement.
NEW ZEALAND, an isolated archipelago located some 1500km east of Australia in the south-west Pacific, was one of the last substantial landmasses to be settled by humans. While the Polynesian ancestors of the indigenous Maori people arrived on the North and South Islands over 1000 years ago, the European settlement of New Zealand occurred much later. Although there was intermittent contact and trade between Europeans and Maori from the late eighteenth century, it was not until the 1830s that the British began to assert their presence in New Zealand, mainly through missionary efforts to 'convert' Maori to Christianity, led by the London-based Church Missionary Society. In February 1840 the Treaty of Waitangi was signed between the British Crown and Maori tribes whereby, and despite serious ambiguities between the intent and meaning of the Maori and English language versions of the Treaty, the British assumed sovereignty over the country and declared New Zealand a Crown colony. Massive British and European immigration immediately ensued and within a decade war over land and issues of sovereignty broke out. For much of the remainder of the nineteenth century, the colony of New Zealand was a war-zone with British colonial forces embroiled in a bitter contest with indigenous Maori for control of large parts of the country. Eventually, the British declared themselves victorious and as punishment, much of the remaining land occupied by Maori tribes was confiscated, following punitive models that had been applied by the British in subduing the 'rebel' Irish clans in the seventeenth century. In addition, successive New Zealand colonial governments worked to redefine, control and ultimately transform the meaning as well as the ownership of Maori land through a series of complex legislative instruments. In this way, almost all of the Maori land estate had passed out of indigenous hands by the early years of the twentieth century. Today, just a small fraction of the original Maori estate remains in Maori ownership (1).
The colonial land surveyors were critical to this history of contest and transformation. Located among the vanguard of European settlers to New Zealand, land surveyors were charged with creating new outposts of empire that would replicate the values, attitudes, and aspirations of the Old World. Moreover, the work of the land surveyors reflects much that is central to the European history of New Zealand, particularly the transformation, domestication and 'taming' of the natural environment. Physically located on the margins of settler society, the land surveyors occupied a central role in implementing colonisation on the ground. They were therefore both boundary markers as well as boundary makers.
The task of the colonial land surveyor was a significant one. Charged with reining in the wilderness and creating order and sense in 'uncharted' territory, the land surveyors occupied a peculiar position in the practical implementation of colonial policy. The lines inscribed by the land surveyors—in maps, drawings and plans as well as on the land itself—were symbols of power and portents of the political, social and economic change that was to follow in their wake. In this way, land surveying was fundamental to the British acquisition of new territory, and represented, in a very graphic and visible way, the thrust of the broader colonial project.
Land surveying in colonial New Zealand had its genesis of course in a much older tradition. In the eighteenth and nineteenth centuries, the expansion of empires, along with the consequent need to delineate national boundaries and construct maps of territorial possession, demanded increasingly accurate methods of land demarcation and measurement. In addition, the process of enclosure and the increasing value of private property in rapidly developing urban centres brought surveying and surveyors into the economic as well as the political arena. By the middle of the nineteenth century, in Europe at least, land survey methods comprised a combination of perspectives and competing practices. While the influence of military techniques (inherited from the Roman architects of land surveying) remained strong, this was accompanied in the latter half of the nineteenth century by a new emphasis on scientific engineering as a direct consequence of British industrialisation.
Broadly speaking, land surveys in colonial contexts fell into two categories. The first involved confirming an already existing cadastre and provided a retrospective vision of land already settled; this was the case where the settlement patterns had long been shaped by history and tradition. The second was concerned with the future rather than the past; focussed on providing a framework for future resettlement. This latter method, what might be termed the 'survey of the future', was enthusiastically adopted in New Zealand, where the land surveyor's chief task was to layer a new spatial order on the existing landscape. Within this 'future' survey, there were two forms: those where the free selection of land parcels had commenced prior to actual settlement and those where settlement was consciously planned in advance. The 'free selection survey' was the most frequent type of survey in colonial New Zealand, at least in the early years of organised settlement. This allowed individuals to determine the boundaries of their own allotment and frequently led to irregular and odd-shaped parcels of land (along with the consequent administrative confusion). On the other hand, surveyors who determined the shape of sections in advance of formal occupation usually laid out a planned settlement for clients such as individual landowners, a company or the government. Free selection prior to survey can still be seen in the design of many rural sections, while the New Zealand Company settlements (the New Zealand city of New Plymouth, for example) are perhaps the best examples of the planned settlement surveys. In their work, New Zealand colonial land surveyors employed instruments which were common to surveying practice elsewhere, including the theodolite, the circumferentor or surveyors' compass, and the prismatic compass (2).
Francis Edward Nairn, 'Mr Mantell at Moeraki ', ink on sketchbook. E333-084-3, Alexander Turnbull Library, National Library of New Zealand/Te Puna Matarauranga o Aotearoa.
So who were these men and how might they best be described? The land surveyors in colonial New Zealand worked in a number of different capacities. From 1854 surveyors were included on the staff of the Land Purchase Department (incorporated into the Native Department in 1885), established to manage the acquisition of Maori land. Until 1862 it was possible to obtain work as a land surveyor without registration, although surveyors under contract to the Crown to survey Crown lands (and so-called 'waste lands') were required to satisfy the standards set by the provincial chief surveyors. In the absence of a standardised system of registration, any person with a minimum knowledge of surveying could practice with minimal experience or qualifications. Consequently, many young men turned to surveying as a relatively easy source of income and adventure.
Following the signing of the Treaty of Waitangi and the assumption of British sovereignty, increasing numbers of British settlers arrived in New Zealand. With expectations about the superior quality and potential of the land, the pressure for land quickly intensified and the demand for surveyors consequently grew. During the 1840s, tension between the nascent colonial government and the New Zealand Company over disparate land policies, together with the demands of Company settlers, who felt cheated of their purchases, stretched the existing resources of both the New Zealand Company and Crown survey services. Sons of missionaries and traders, many of whom were educated in mission schools and were fluent in the Maori language, were readily recruited as surveyors. It is worth noting that their intimate knowledge of Maori culture and language proved invaluable not only in negotiating the survey, but also in facilitating the purchase of much of the Maori land estate. With the further influx of settlers in the 1850s and 1860s, the colonial government increasingly looked to the immigrant population for additional survey staff.
Most but by no means all of the early surveyors were born in England and made New Zealand their adoptive home. Some came from Scotland and Ireland, while others came from further afield, particularly central Europe. A large number of land surveyors had spent time in the Australian colonies before arriving in New Zealand; indeed, most of the early surveyors who have left records had travelled extensively before arriving in the colony. Some had left established careers to immigrate and brought experience with them, while others were young men who chose to 'cut their teeth' in developing a surveying career in New Zealand. Robert Park and John Rochfort, for instance, were trained in engineering; Charles Heaphy initially trained as a draughtsman; and Theophilus Heale, later chief surveyor and inspector of surveys, was educated as a classical scholar, mathematician and navigator. Some—Charles Heaphy, Samuel Brees and John Buchanan, for instance—were highly accomplished artists, while others had trained as draughtsmen and drew on these technical skills in the course of their surveying fieldwork. Others became active in political roles, at both the provincial and national level. Frederick Carrington, for example, was a provincial superintendent and a Member of the New Zealand Parliament. Land surveying was often a natural choice of occupation for those young male settlers who could make use of their skills in a particularly practical fashion. In addition, the colonial land surveyors were often men of learning and intellectual ability, with interests in poetry, ethnology, philology and geology. It would be fair to say too that land surveying, due to its physical rigors, typically attracted young men with a keen sense of adventure and an abundance of energy. For many young men, a surveying career also promised the challenge of working on the colonial frontier (3).
The everyday reality for the colonial land surveyor was clearly challenging and probably sat somewhere between these highly romanticised images. The rain, 'the bog' and the inclement weather elicited frequent comments and complaints. Generally, the climatic challenges, while inconvenient, were seen as a test of strength and fortitude. 'The [New Zealand] Company's surveyors whose life is almost wholly spent in the bush,' Charles Heaphy remarked in 1842, 'and who often pursue their vocation in all weathers, are amongst the healthiest and most robust men in the colony.' (6) They had to be able-bodied and strong, as living arrangements were often makeshift, temporary and haphazard, and often fraught with risk.
Despite the aggressive pace of the British colonisation of New Zealand post-1840, most of the early colonial land surveyors were keenly aware that they were not 'first-time' explorers, but were traversing landscapes that were already known, named and mapped by indigenous Maori. Certainly, from the 1840s through to the 1860s the land surveyors' efforts to name, tame and claim the land were much less an exercise in possessing it outright and more about transitioning the meaning of land (its value, attributes and qualities) from one cultural framework to another. From the 1860s, and especially with the operation of the Native Land Court from 1865 onwards, the work of land surveyors took on a more formidable role as Maori land was transferred from collective customary tenure to individual Crown-derived titles and, in many cases, was permanently alienated. This meant that land surveyors were in frequent contact and negotiation with resident Maori communities; indeed, for many communities, the land surveyor was the 'face' of the new colonial order.
Maori responded to the increased demand for their services from the land surveyors by adapting and expanding their existing economic networks. Arthur Dobson, for instance, wrote that on the West Coast of the South Island, 'as time went on able-bodied young men that I had working for me sent word to the various pas (Maori village settlements) down the coast that I was coming, and that I would pay for help for canoeing on the rivers'.(8) When Dobson arrived to commence his survey, members of the local tribe Ngai Tahu, were ready and waiting for business.
The negotiation of this sort of boundary making eventually cut both ways, however, and Maori opposition to surveying was not uncommon as lines were drawn, often arbitrarily, through cultivations and across tribal boundaries. Indeed, from the 1860s onwards, for many Maori communities the presence of the land surveyor became a metaphor for loss and a portent of impending land alienation. According to land surveyors who worked in Taranaki [in the southern North Island], Maori frequently (and publicly) demonstrated their opposition to the conduct of land surveys. While laying out the settlement of New Plymouth, a New Zealand Company settlement in the lower North Island Taranaki region during the early months of 1841, Frederick Carrington was confronted by 'natives from the interior who said we that we should not cut any more. They flourished their tomahawks, and danced and yelled, and I thought we should all be massacred.'(10) In Taranaki, this reaction was not surprising, given that many of the purchases were highly contested, both at the time, and later, in the form of submissions to successive government commissions of inquiry. Tensions between the Ngati Toa tribe and the New Zealand Company land surveyors working at Wairau, near Nelson in the north of the South Island, reached a climax in June 1843, when 22 settlers and six Maori were killed. The incident followed an attempt by officials of the New Zealand Company in Nelson to seize by force land from the great rangatira (chief) Te Rauparaha, who denied having sold the land.
For Maori, the surveyor's theodolite—commonly referred to as the 'taipo' or 'tipo'—was also a symbol of uncertainty and possible conflict. From the 1840s, the erection of survey poles, like the traditional pou whenua marker-poles of Maori society, signified an explicit and aggressive act of possession. Maori leaders therefore often regarded the intrusion of the surveyors and their boundary markers as overt challenges to their mana. While surveying Ngai Tahu land reserves in the South Island in September 1848, the surveyor Walter Mantell noted how '[t]wo or three old men not understanding the erection of a pole at their huts at Waitueri threw it away with the others which the man carried. I went down [and] lectured them [and] explained the use of the pole and remained there.'(11) There is much evidence to suggest that Maori well understood the erection of the survey poles, and their removal was a deliberate act of protest at Mantell's marking out of the reserves. The surveyor Edwin Brookes cited the suspicion Taranaki Maori held towards the theodolite in the 1870s: 'The invariable expression that would come over them after a long drawn breath was "taipo", meaning evil spirit: by my interpretation was—a mystery, or something mysterious. In order to show them a friendly spirit, I would allow many of these natives to look through the telescope, when they would withdraw from it much perplexed.'(12) While impressed by the technology, Maori no doubt appreciated the powerful role of the theodolite in the survey and alienation of their lands.
These examples of conflict serve both to illustrate the precarious position occupied by the early land surveyors working in the field and highlight how an indigenous Maori system of naming and mapping pre-dated and indeed co-existed with the new order imposed by the colonial land surveyors. For Maori, boundaries on the land formed the basis of an indigenous system of mapping. As the basis of tribal economy and community life, land was identified through a complex system of rights and privileges that relied on physical as well as cultural boundary markers. While whakapapa (genealogical) connections, waiata (song), and 'mental maps' were used in navigating the land, boundaries were indicated by geographical features such as hills, rock formations and rivers. Stones, wooden posts and holes dug into the ground also functioned as markers between tribal areas, and individual cultivation plots were often the most enduring divisional marks. Maori also diverted streams and constructed estuarine canals to assist with fishing and to act as boundary markers. Prior to organised British settlement in the mid nineteenth century, there was little need to precisely delineate physical boundaries. From that point forward, however, boundaries on the land became symbols of identity, 'ownership' and esteem; established and legitimized in public and official discourse and given popular currency by government legislation. Western capitalist ideas of land tenure and individual property ownership then quickly dominated. The land wars, and their issue—the Native Land Courts and the raupatu—also played a role in this fundamental shift in thinking. While indigenous Maori perceptions of land use and ownership continued, European (and especially British) ideals about land usage and administration soon became the norm rather than the exception.
Land surveying was fundamental to the British colonizing vision and the acquisition of new territories, such as New Zealand, for settlement. The work of the colonial land surveyors reflects much that is central to the history of New Zealand, particularly the transformation and domestication of the natural environment. The land surveyors were crucial in effecting change, as they occupied a central role in implementing the principles of colonization on the ground: they operated, quite literally, at the 'cutting edge' of colonization. Their work deserves, therefore, to be remembered, though it also needs to be understood in context. While it would be tempting to oversimplify the contribution of the land surveyors to New Zealand's past, and (depending on your point of view) to either valorize or demonize their work as colonial entrepreneurs, it is worth remembering that the land surveyors were agents of their time who were capable of questioning the longer-term outcomes as well as the immediate impact of their work. We need to remember them as complex historical actors whose work has shaped the contours of our historical trajectory and fashioned our modern society in ways more powerful than we fully realize.
Land alienation, in addition to a range of other issues, is the subject of the vast majority of claims by modern Maori to the Waitangi Tribunal, a commission of inquiry established in 1975 to inquire into the allegations by Maori tribes that the Crown has consistently failed to honor its responsibilities under the 1840 Treaty of Waitangi. See further, Claudia Orange, The Treaty of Waitangi, Allen and Unwin, Wellington, 1987; Alan Ward, An Unsettled History: Treaty Claims in New Zealand Today, Bridget Williams Books, Wellington, 1998; Giselle Byrnes, The Waitangi Tribunal and New Zealand History, Oxford University Press, Melbourne, 2004.
Surveying in North America embraced both types of survey, while in the Australian colonies, planned rectilinear (or equal square) land division was the most common practice. Chain surveying was also a common practice, where the Gunter's chain, 66-foot long and divided into equal links, was used for calculating distance.
John Rochfort, The Adventures of a Surveyor in New Zealand and the Australian Gold Diggings, London, 1853.
Edward Jerningham Wakefield, Adventure in New Zealand, first edition John Murray (ed.), London, 1845, revised edition Joan Stevens (ed.), Auckland, 1975, pp. 233-34.
John Turnbull Thomson, 'Extracts from a journal kept during the reconnaissance survey of the southern districts of the province of Otago', in Nancy Taylor (ed.), Early Travellers in New Zealand, London, 1959, p. 347. See also John Turnbull Thomson, MS-Papers-0176, Alexander Turnbull Library, Wellington, and John Turnbull Thomson, Rambles with a Philosopher, Dunedin, 1867.
Charles Heaphy, Narrative of a Residence in Various Parts of New Zealand, London, 1842, p. 23.
Rochfort, The Adventures of a Surveyor; Jollie wrote 'this old Maori was named after Sir George Grey the Governor—twice—of New Zealand, by I believe, Mr Walter Mantel [sic].' Edward Jollie, Reminiscences 1825-94, MS-Papers-4207, Alexander Turnbull Library, Wellington, p. 27. Similarly, the wives of Kehu and Pikewate joined them in guiding Thomas Brunner down the West Coast of the South Island on his 'Great Journey' of discovery in 1846-48.
Cited in Arthur Dudley Dobson, Reminiscences of Arthur Dudley Dobson, Engineer, 1840-1930, Christchurch, 1930.
Henry Reynolds, 'The land, the explorers and the Aborigines', Historical Studies, vol. 19, no. 5 (1980), pp. 213-26; Paul Carter, The Road to Botany Bay: an essay in spatial history, London, 1987, p. 340.
F. A. Carrington, cited in William H. J. Seffern, Chronicles of the Garden of New Zealand Known as Taranaki, New Plymouth, 1896, p. 47.
Walter Mantell, 'Journal Kaiapoi to Otago, 1848-49', MS-Papers-1543, Alexander Turnbull Library, Wellington.
Edwin Brookes, Frontier Life: Taranaki, New Zealand, Auckland, 1892, pp. 38-39.
The confiscation of Maori land, or raupatu, was ushered in under the rather euphemistically titled 'The New Zealand Settlements Act 1863'.
As Nola Easdale has shown in New Zealand and Stephen Martin for Australia, surveyors' diaries and field books are particularly rich historical sources. See further Nola Easdale, Kairuri, the measurer of land: the life of the nineteenth century surveyor pictured in his art and his writings, Highgate/Price Milburn, Petone, 1988; Stephen Martin, A New Land: European perceptions of Australia, 1788-1850, Allen & Unwin, St Leonards, New South Wales, 1993.
Professor Giselle Byrnes is Pro Vice-Chancellor of the Faculty of Law, Education, Business and Arts at Charles Darwin University. Professor Byrnes moved to the Northern Territory from New Zealand in mid 2011 with her family to take up this role. She was formerly Professor of History and Pro Vice-Chancellor (Postgraduate) at the University of Waikato, Hamilton, New Zealand (2007-2011).
Giselle Byrnes completed a PhD in History at the University of Auckland in the mid-1990s and then worked as an historian for the Waitangi Tribunal. She returned to academia in 1997 and taught in the Department of History at Victoria University of Wellington for a ten-year period (1997-2007). Here she established the Public History programme and developed a suite of courses in social and cultural history, with a particular emphasis on exploring history in colonial and postcolonial contexts.
Professor Byrnes' publications include Boundary Markers: Land Surveying and the Colonisation of New Zealand (2001), The Waitangi Tribunal and New Zealand History (2004) and The New Oxford History of New Zealand (2009), for which she was Editor. She has also published numerous articles on various aspects of colonial, settler and Indigenous histories, in addition to public history. While the focus of her work has been grounded in New Zealand historical experiences, she has expertise in comparative colonial and transnational historical methodologies.
In 2006 Giselle Byrnes was Fulbright Visiting Professor in New Zealand Studies at Georgetown University, Washington DC, and she has served a term as National President of the New Zealand Historical Association.
Visit Norfolk Island and experience the historic site together with FIG International Institution for the History of Surveying and Measurement, who invites you to Charting and Mapping the Pacific Paradise of the Pitcairners Conference The conferences will be held at Norfolk Island, New Zealand, 6-10 July 2014.


text:
function [mach] = ftps2mach(ftps)
% Convert speed from feet per second to mach number (at STP!)
% Chad A. Greene 2012
mach = ftps*0.0008886297376093 ;

redpajama_set_name: RedPajamaGithub


text:
This is a list of electricity-generating power stations in the U.S. state of South Carolina, sorted by type and name. In 2020, South Carolina had a total summer capacity of 24,122 MW through all of its power plants, and a net generation of 98,529 GWh. The corresponding electrical energy generation mix in 2021 was 53.8% nuclear, 23.3% natural gas, 15.2% coal, 4% hydroelectric, 2.3% biomass, 2.3% solar, and 0.1% petroleum.

South Carolina is the nation's third largest producer of nuclear power, with four nuclear plants. Natural gas has been the most rapidly growing source of generation; quadrupling over the decade starting 2010. Data from the U.S. Energy Information Administration serves as a general reference.

Nuclear power stations

Fossil-fuel power stations

Coal

 Units 2 and 3 were originally planned to be converted to natural gas as an interim step to closure, but those plans were abandoned.
 Fueled by mix of coal and natural gas
 Units 3 and 4 are permitted to fire up to 30% petcoke by weight on either boiler.

Natural Gas

Renewable power stations

Biomass

Hydroelectric

Solar

Storage power stations

Pumped storage

See also

List of power stations in the United States
List of pumped-storage hydroelectric power stations

References

South Carolina
Lists of buildings and structures in South Carolina
Energy in South Carolina

redpajama_set_name: RedPajamaWikipedia


text:
package main // import "github.com/Jimdo/asg-ebs"

import (
	"errors"
	"fmt"
	"io/ioutil"
	"os"
	"os/exec"
	"sort"
	"strings"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
	"github.com/aws/aws-sdk-go/aws/ec2metadata"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/ec2"

	"gopkg.in/alecthomas/kingpin.v2"

	log "github.com/Sirupsen/logrus"
)

type createFileSystemOnVolumeTimeout struct{}

func (e createFileSystemOnVolumeTimeout) Error() string {
	return "Volume Timeout"
}

type ByStartTime []*ec2.Snapshot

func (s ByStartTime) Len() int           { return len(s) }
func (s ByStartTime) Swap(i, j int)      { s[i], s[j] = s[j], s[i] }
func (s ByStartTime) Less(i, j int) bool { return (*s[i].StartTime).Before(*s[j].StartTime) }

func waitForFile(file string, timeout time.Duration) error {
	startTime := time.Now()
	if _, err := os.Stat(file); err == nil {
		return nil
	}
	newTimeout := timeout - time.Since(startTime)
	if newTimeout > 0 {
		return waitForFile(file, newTimeout)
	} else {
		return errors.New("File " + file + " not found")
	}
}

func run(cmd string, args ...string) error {
	log.WithFields(log.Fields{"cmd": cmd, "args": args}).Info("Running command")
	out, err := exec.Command(cmd, args...).CombinedOutput()
	if err != nil {
		log.WithFields(log.Fields{"cmd": cmd, "args": args, "err": err, "out": string(out)}).Info("Error running command")
		return err
	}
	return nil
}

func slurpFile(file string) string {
	v, err := ioutil.ReadFile(file)
	if err != nil {
		log.WithFields(log.Fields{"err": err, "file": file}).Info("Failed to read file")
	}
	return string(v)
}

type AsgEbs interface {
	checkDevice(device string) error
	checkMountPoint(mountPoint string) error
	findVolume(tagKey string, tagValue string) (*string, error)
	attachVolume(volumeId string, attachAs string, deleteOnTermination bool) error
	findSnapshot(tagKey string, tagValue string) (*string, error)
	createVolume(createSize int64, createName string, createVolumeType string, createTags map[string]string, snapshotId *string) (*string, error)
	mountVolume(device string, mountPoint string) error
	makeFileSystem(device string, mkfsInodeRatio int64, volumeId string) error
	waitUntilVolumeAvailable(volumeId string) error
}

type AwsAsgEbs struct {
	AwsConfig        *aws.Config
	Region           string
	AvailabilityZone string
	InstanceId       string
}

func NewAwsAsgEbs(maxRetries int) *AwsAsgEbs {
	awsAsgEbs := &AwsAsgEbs{}

	metadata := ec2metadata.New(session.New())

	region, err := metadata.Region()
	if err != nil {
		log.WithFields(log.Fields{"error": err}).Fatal("Failed to get region from instance metadata")
	}
	log.WithFields(log.Fields{"region": region}).Info("Setting region")
	awsAsgEbs.Region = region

	availabilityZone, err := metadata.GetMetadata("placement/availability-zone")
	if err != nil {
		log.WithFields(log.Fields{"error": err}).Fatal("Failed to get availability zone from instance metadata")
	}
	log.WithFields(log.Fields{"az": availabilityZone}).Info("Setting availability zone")
	awsAsgEbs.AvailabilityZone = availabilityZone

	instanceId, err := metadata.GetMetadata("instance-id")
	if err != nil {
		log.WithFields(log.Fields{"error": err}).Fatal("Failed to get instance id from instance metadata")
	}
	log.WithFields(log.Fields{"instance_id": instanceId}).Info("Setting instance id")
	awsAsgEbs.InstanceId = instanceId

	awsAsgEbs.AwsConfig = aws.NewConfig().
		WithRegion(region).
		WithCredentials(ec2rolecreds.NewCredentials(session.New())).
		WithMaxRetries(maxRetries)

	return awsAsgEbs
}

func (awsAsgEbs *AwsAsgEbs) findVolume(tagKey string, tagValue string) (*string, error) {
	svc := ec2.New(session.New(awsAsgEbs.AwsConfig))

	params := &ec2.DescribeVolumesInput{
		Filters: []*ec2.Filter{
			{
				Name: aws.String("tag:" + tagKey),
				Values: []*string{
					aws.String(tagValue),
				},
			},
			{
				Name: aws.String("tag:filesystem"),
				Values: []*string{
					aws.String("true"),
				},
			},
			{
				Name: aws.String("status"),
				Values: []*string{
					aws.String("available"),
				},
			},
			{
				Name: aws.String("availability-zone"),
				Values: []*string{
					aws.String(awsAsgEbs.AvailabilityZone),
				},
			},
		},
	}

	describeVolumesOutput, err := svc.DescribeVolumes(params)
	if err != nil {
		return nil, err
	}
	if len(describeVolumesOutput.Volumes) == 0 {
		return nil, nil
	}
	return describeVolumesOutput.Volumes[0].VolumeId, nil
}

func (awsAsgEbs *AwsAsgEbs) findSnapshot(tagKey string, tagValue string) (*string, error) {
	svc := ec2.New(session.New(awsAsgEbs.AwsConfig))

	describeSnapshotsInput := &ec2.DescribeSnapshotsInput{
		Filters: []*ec2.Filter{
			{
				Name: aws.String("tag:" + tagKey),
				Values: []*string{
					aws.String(tagValue),
				},
			},
			{
				Name: aws.String("status"),
				Values: []*string{
					aws.String("completed"),
				},
			},
		},
	}
	describeSnapshotsOutput, err := svc.DescribeSnapshots(describeSnapshotsInput)
	if err != nil {
		return nil, err
	}
	snapshots := describeSnapshotsOutput.Snapshots
	sort.Sort(sort.Reverse(ByStartTime(snapshots)))

	if len(snapshots) == 0 {
		return nil, nil
	}

	return snapshots[0].SnapshotId, nil
}

func (awsAsgEbs *AwsAsgEbs) createVolume(createSize int64, createName string, createVolumeType string, createTags map[string]string, snapshotId *string) (*string, error) {
	svc := ec2.New(session.New(awsAsgEbs.AwsConfig))

	filesystem := "false"

	createVolumeInput := &ec2.CreateVolumeInput{
		AvailabilityZone: &awsAsgEbs.AvailabilityZone,
		Size:             aws.Int64(createSize),
		VolumeType:       aws.String(createVolumeType),
	}

	if snapshotId != nil {
		createVolumeInput.SnapshotId = aws.String(*snapshotId)
		filesystem = "true"
	}

	vol, err := svc.CreateVolume(createVolumeInput)
	if err != nil {
		return nil, err
	}
	tags := []*ec2.Tag{
		{
			Key:   aws.String("Name"),
			Value: aws.String(createName),
		},
		{
			Key:   aws.String("filesystem"),
			Value: aws.String(filesystem),
		},
	}
	for k, v := range createTags {
		tags = append(tags,
			&ec2.Tag{
				Key:   aws.String(k),
				Value: aws.String(v),
			},
		)
	}

	createTagsInput := &ec2.CreateTagsInput{
		Resources: []*string{vol.VolumeId},
		Tags:      tags,
	}
	_, err = svc.CreateTags(createTagsInput)
	if err != nil {
		return vol.VolumeId, err
	}

	return vol.VolumeId, nil
}

func (awsAsgEbs *AwsAsgEbs) waitUntilVolumeAvailable(volumeId string) error {
	svc := ec2.New(session.New(awsAsgEbs.AwsConfig))

	describeVolumeInput := &ec2.DescribeVolumesInput{
		VolumeIds: []*string{aws.String(volumeId)},
	}
	err := svc.WaitUntilVolumeAvailable(describeVolumeInput)
	if err != nil {
		return &createFileSystemOnVolumeTimeout{}
	}
	return nil
}

func (awsAsgEbs *AwsAsgEbs) attachVolume(volumeId string, attachAs string, deleteOnTermination bool) error {
	svc := ec2.New(session.New(awsAsgEbs.AwsConfig))

	attachVolumeInput := &ec2.AttachVolumeInput{
		VolumeId:   aws.String(volumeId),
		Device:     aws.String(attachAs),
		InstanceId: aws.String(awsAsgEbs.InstanceId),
	}
	_, err := svc.AttachVolume(attachVolumeInput)
	if err != nil {
		return err
	}

	describeVolumeInput := &ec2.DescribeVolumesInput{
		VolumeIds: []*string{aws.String(volumeId)},
	}
	err = svc.WaitUntilVolumeInUse(describeVolumeInput)
	if err != nil {
		return err
	}

	if deleteOnTermination {
		modifyInstanceAttributeInput := &ec2.ModifyInstanceAttributeInput{
			Attribute:  aws.String("blockDeviceMapping"),
			InstanceId: aws.String(awsAsgEbs.InstanceId),
			BlockDeviceMappings: []*ec2.InstanceBlockDeviceMappingSpecification{
				{
					DeviceName: aws.String(attachAs),
					Ebs: &ec2.EbsInstanceBlockDeviceSpecification{
						DeleteOnTermination: aws.Bool(true),
						VolumeId:            aws.String(volumeId),
					},
				},
			},
		}
		_, err = svc.ModifyInstanceAttribute(modifyInstanceAttributeInput)
		if err != nil {
			return err
		}
	}

	err = waitForFile("/dev/"+attachAs, 60*time.Second)
	if err != nil {
		return err
	}

	return nil
}

func (awsAsgEbs *AwsAsgEbs) makeFileSystem(device string, mkfsInodeRatio int64, volumeId string) error {
	svc := ec2.New(session.New(awsAsgEbs.AwsConfig))

	err := run("/usr/sbin/mkfs.ext4", "-i", fmt.Sprintf("%d", mkfsInodeRatio), device)
	if err != nil {
		return err
	}
	tags := []*ec2.Tag{
		{
			Key:   aws.String("filesystem"),
			Value: aws.String("true"),
		},
	}
	createTagsInput := &ec2.CreateTagsInput{
		Resources: []*string{aws.String(volumeId)},
		Tags:      tags,
	}
	_, err = svc.CreateTags(createTagsInput)
	if err != nil {
		return err
	}
	return nil
}

func (awsAsgEbs *AwsAsgEbs) mountVolume(device string, mountPoint string) error {
	err := os.MkdirAll(mountPoint, 0755)
	if err != nil {
		return err
	}
	return run("/bin/mount", device, mountPoint)
}

func (awsAsgEbs *AwsAsgEbs) checkDevice(device string) error {
	if _, err := os.Stat(device); !os.IsNotExist(err) {
		return errors.New("Device exists")
	}
	return nil
}

func (awsAsgEbs *AwsAsgEbs) checkMountPoint(mountPoint string) error {
	if strings.Contains(slurpFile("/proc/mounts"), mountPoint) {
		return errors.New("Already mounted")
	}
	return nil
}

type CreateTagsValue map[string]string

func (v CreateTagsValue) Set(str string) error {
	parts := strings.SplitN(str, "=", 2)
	if len(parts) != 2 {
		return fmt.Errorf("expected KEY=VALUE got '%s'", str)
	}
	key := parts[0]
	value := parts[1]
	v[key] = value
	return nil
}

func (v CreateTagsValue) String() string {
	return ""
}

func CreateTags(s kingpin.Settings) (target *map[string]string) {
	newMap := make(map[string]string)
	target = &newMap
	s.SetValue((*CreateTagsValue)(target))
	return
}

func runAsgEbs(asgEbs AsgEbs, cfg Config) {

	createFileSystemOnVolume := false
	var volumeId *string
	var snapshotId *string
	attachAsDevice := "/dev/" + *cfg.attachAs

	// Precondition checks
	err := asgEbs.checkDevice(attachAsDevice)
	if err != nil {
		log.WithFields(log.Fields{"device": attachAsDevice}).Fatal("Device already exists")
	}

	err = asgEbs.checkMountPoint(*cfg.mountPoint)
	if err != nil {
		log.WithFields(log.Fields{"mount_point": *cfg.mountPoint}).Fatal("Already mounted")
	}

	if *cfg.snapshotName == "" {
		for i := 1; i <= 10; i++ {
			volumeId, err = asgEbs.findVolume(*cfg.tagKey, *cfg.tagValue)
			if err != nil {
				log.WithFields(log.Fields{"error": err}).Fatal("Failed to find volume")
			}
			if volumeId == nil {
				break
			} else {
				log.WithFields(log.Fields{"volume": *volumeId, "device": attachAsDevice, "attempt": i}).Info("Trying to attach existing volume")
				err = asgEbs.attachVolume(*volumeId, *cfg.attachAs, *cfg.deleteOnTermination)
				if err != nil {
					log.WithFields(log.Fields{"error": err}).Warn("Failed to attach volume")
				} else {
					break
				}
			}
		}
	} else {
		snapshotId, err = asgEbs.findSnapshot("Name", *cfg.snapshotName)
		if err != nil {
			log.WithFields(log.Fields{"error": err, "snapshot_name": *cfg.snapshotName}).Fatal("Failed to find snapshot")
		}
	}

	if volumeId == nil {
		log.Info("Creating new volume")
		volumeId, err = asgEbs.createVolume(*cfg.createSize, *cfg.createName, *cfg.createVolumeType, *cfg.createTags, snapshotId)
		if err != nil {
			log.WithFields(log.Fields{"error": err}).Fatal("Failed to create new volume")
		}
		log.WithFields(log.Fields{"volume": *volumeId}).Info("Waiting until new volume is available")
		err = asgEbs.waitUntilVolumeAvailable(*volumeId)
		if err != nil {
			log.WithFields(log.Fields{"error": err, "volume": *volumeId}).Fatal("Waiting for volume timed out")
		}
		if snapshotId == nil {
			createFileSystemOnVolume = true
		}
		log.WithFields(log.Fields{"volume": *volumeId, "device": attachAsDevice}).Info("Attaching volume")
		err = asgEbs.attachVolume(*volumeId, *cfg.attachAs, *cfg.deleteOnTermination)
		if err != nil {
			log.WithFields(log.Fields{"error": err}).Fatal("Failed to attach volume")
		}
	}

	if createFileSystemOnVolume {
		log.WithFields(log.Fields{"device": attachAsDevice}).Info("Creating file system on new volume")
		err = asgEbs.makeFileSystem(attachAsDevice, *cfg.mkfsInodeRatio, *volumeId)
		if err != nil {
			log.WithFields(log.Fields{"error": err}).Fatal("Failed to create file system")
		}
	}

	log.WithFields(log.Fields{"device": attachAsDevice, "mount_point": *cfg.mountPoint}).Info("Mounting volume")
	err = asgEbs.mountVolume(attachAsDevice, *cfg.mountPoint)
	if err != nil {
		log.WithFields(log.Fields{"error": err}).Fatal("Failed to mount volume")
	}

}

type Config struct {
	tagKey              *string
	tagValue            *string
	attachAs            *string
	mountPoint          *string
	createSize          *int64
	mkfsInodeRatio      *int64
	createName          *string
	createVolumeType    *string
	createTags          *map[string]string
	deleteOnTermination *bool
	snapshotName        *string
	maxRetries          *int
}

func main() {
	cfg := &Config{
		tagKey:              kingpin.Flag("tag-key", "The tag key to search for").Required().PlaceHolder("KEY").String(),
		tagValue:            kingpin.Flag("tag-value", "The tag value to search for").Required().PlaceHolder("VALUE").String(),
		attachAs:            kingpin.Flag("attach-as", "device name e.g. xvdb").Required().PlaceHolder("DEVICE").String(),
		mountPoint:          kingpin.Flag("mount-point", "Directory where the volume will be mounted").Required().PlaceHolder("DIR").String(),
		createSize:          kingpin.Flag("create-size", "The size of the created volume, in GiBs").Required().PlaceHolder("SIZE").Int64(),
		mkfsInodeRatio:      kingpin.Flag("mkfs-inode-ratio", "mkfs.ext4 inode ratio (-i)").Default("16384").Int64(),
		createName:          kingpin.Flag("create-name", "The name of the created volume").Required().PlaceHolder("NAME").String(),
		createVolumeType:    kingpin.Flag("create-volume-type", "The volume type of the created volume. This can be `gp2` for General Purpose (SSD) volumes or `standard` for Magnetic volumes").Required().PlaceHolder("TYPE").Enum("standard", "gp2"),
		createTags:          CreateTags(kingpin.Flag("create-tags", "Tag to use for the new volume, can be specified multiple times").PlaceHolder("KEY=VALUE")),
		deleteOnTermination: kingpin.Flag("delete-on-termination", "Delete volume when instance is terminated").Bool(),
		snapshotName:        kingpin.Flag("snapshot-name", "Name of snapshot to use for new volume").String(),
		maxRetries:          kingpin.Flag("max-retries", "Maximum number of retries for AWS requests").Default("20").Int(),
	}

	kingpin.UsageTemplate(kingpin.CompactUsageTemplate)
	kingpin.CommandLine.Help = "Script to create, attach, format and mount an EBS Volume to an EC2 instance"
	kingpin.Parse()

	awsAsgEbs := NewAwsAsgEbs(*cfg.maxRetries)

	runAsgEbs(awsAsgEbs, *cfg)

}


text:
Q: Ruby returning integer-only when item is in an array I'm creating a reverse polish notation calculator as a 'warm-up test' for school.  I've almost nailed it.  The issue I'm coming across is when I run this myself, I see just the integer returned (which is desired).  When I plug this into the school's RSpec checker it returns my data in an array, so it's flagged as incorrect.
To fix, I just built the queue.each statement on the end.  I tried this in a few different positions, but it does not seem to matter.  Is there a bigger concept to extract my answer out of the array format when evaluate returns an answer?
class RPNCalculator

    def evaluate(string)
        holding = []
        queue = []
        string = string.split(" ").to_a      #jam string into an array - break on spaces

        string.each do |x|
            if x.match(/\d/)      #number checker. if true throw it in queue.
             queue << x.to_i       #convert to number from string
            elsif x.match(/\+/)      #math operator checker
                holding << queue[-2] + queue[-1]     #grab queue's last 2 & replace w/ result
                queue.pop(2)
                queue << holding[-1]
            elsif x.match(/\-/)
                holding << queue[-2] - queue[-1]
                queue.pop(2)
                queue << holding[-1]
            elsif x.match(/\*/)
                holding << queue[-2] * queue[-1]
                queue.pop(2)
                queue << holding[-1]
            else
            end
        end
            return queue.each { |x| puts x.to_i}     # by [-1] in string, queue holds answer
    end
end

Thank you in advance for your time,

A: Your method (without queue.each) is returning the result of string.each.
If you want to return queue, you need to do that.
class RPNCalculator

    def evaluate(string)
        holding = []
        queue = []
        string = string.split(" ").to_a      #jam string into an array - break on spaces

        string.each do |x|
          #...
        end
        queue[0] # return the final result, stored in queue
    end
end

redpajama_set_name: RedPajamaStackExchange


The St. Austins development comprises of 11 luxury apartments. The building was sympathetically restored and converted from a rundown office block into a contemporary development in the heart of the town centre.
The Heaton Group design team maximised all available space, to provide highly sought after, unique, spacious apartments.
Several apartments boast a split mezzanine layout, while all of them offer signature designs with subtle references to the buildings former life as a solicitor's office.
Investor Highlight: Championing the building's history resulted in characterful properteries that were immediately in demand.
Why Did We Choose Warrington?
Based in neighbouring Wigan, The Heaton Group is well aware of the exceptional investment opportunities that can be found in Warrington.
The town is currently benefitting from Northern Powerhouse investment, with transport links allowing access to Manchester and Liverpool in as little as 20 minutes.
Therefore, Warrington is perfectly placed to take advantage of the rising number of young professionals moving into the area.
Investor Highligh: Around 700,000 professionals are living within a 30 minute commute of Warrington.
St. Austins was, and continues to be, a financial success for our investors. With a gross development value of £1,275,000, we were able to offer the apartments in a brilliant, central location at a rental yield of 6.9%.
Property investment isn't just about finding a living space at a cheap price. The quality and character of a property play a fundamental role in increase demand and maximising revenue.
St. Austins was purchased in March 2016 after its history as a solicitor's office came to an end. Once planning permission was obtained in June 2016, construction began to convert the Listed Building into 11 luxury apartments, 9 one-beds and 2 two-beds.
Sympathetically designed to showcase the building's natural features, St. Austins was immediately sort after, with one investor buying and funding the development mid-construction.
Extensive time and care was taken when redeveloping the property to maintain the original features of the Grade II Listed structure but this did not stop our team completing in under 12 months.
Since then, the property has been fully let to working professionals within the private rented sector (PRS). Due to the volume of demand in the property, our property management team now have a waiting list of prospective tenants.
Completed on time and on budget, St. Austins epitomises the type of property that we strive to provide our investors. We're proud to call this a Heaton Group property and be contributing to the wider development of the Warrington area.
If you want to learn more about this project and any we have coming up in the future, book a call with a member of our team.


text:
3rd-strike.com | December's Xbox Games With Gold and Game Pass update
December's Xbox Games With Gold and Game Pass update
December 4, 2018 - Pimp Without The Pee - News - No Comments
As will be known for everybody who ever used Xbox Live Gold (the purchase needed to play online), while you have it activated you get a few new games every month which are then "free to play". This is brought under the name "Games With Gold". The Game Pass is a different subscription you can apply to, granting you access to a library of games that keeps getting updated every now and then. Below the list for both subscriptions their changes in the month of December, 2018:
Only for Xbox One with Xbox Live Gold:
Q.U.B.E 2 – (1st of December up to and including the 31st of December)
Never Alone – (16th of December up to and including the 15th of January)
For Xbox One AND Xbox 360 with Xbox Live Gold:
Dragon Age II – (1st of December up to and including the 15th of December)
Mercenaries: Playground of Destruction – (16th of December up to and including the 31st of December)
The Xbox Game Pass News:
Three new games have been added at different dates. The Gardens Between at the 29th of November, Mutant Year Zero at the 4th of December, and Strange Brigade at the 6th of December.
There now is an Xbox Game Pass app available for Android and iOS. This app gives you updates when new games are available like the one ones named above, but also gives you the option to install games on your Xbox One from a distance, allowing you to instantly play after you get home from your daily destinations.
Falling Out – Review
PC, PS4, PS5, Reviews, Switch, Xbox One
South of the Circle – Review
iPhone/iPad/Android, PC, PS4, PS5, Reviews, Switch, Xbox One, Xbox Series X/S
Experts explain Spongebob SquarePants: The Cosmic Shake

redpajama_set_name: RedPajamaCommonCrawl


text:
Aggressive testing helps Italian town cut new coronavirus cases to zero
Expert urges all countries to ramp up the numbers of people being screened
Rome's Town Hall is lit up with the colours of the Italian flag to show solidarity with the country as it continues to battle coronavirus. Photograph: Reuters/ Alberto Lingria
Tue Mar 17 2020 - 19:37
An infection control experiment that was rolled out in a small Italian community at the start of Europe's coronavirus crisis has stopped all new infections in the town that was at the centre of the country's outbreak.
Through testing and retesting of all 3,300 inhabitants of the town of Vò, near Venice, regardless of whether they were exhibiting symptoms, and rigorous quarantining of their contacts once infection was confirmed, health authorities have been able to completely stop the spread of the illness there.
Andrea Crisanti, an infections expert at Imperial College London who is taking part in the Vò project while on sabbatical at the University of Padua, urged countries that have been limiting virus testing, which includes the UK and US, to learn lessons and ramp up the numbers of people being screened.
"In the UK, there are a whole lot of infections that are completely ignored," Prof Crisanti said. "We were able to contain the outbreak here because we identified and eliminated the 'submerged' infections and isolated them," he said of the Vò approach. "That is what makes the difference."
'Test, test, test'
The success underscores the importance of testing and isolating otherwise healthy carriers, an approach that has been strongly endorsed by the World Health Organisation. The WHO this week urged all countries to test aggressively, noting that South Korea and Taiwan were having success in limiting infections by doing so. "Our key message is: test, test, test," Tedros Adhanom Ghebreyesus, WHO chief, said on Monday.
Prof Crisanti said the unusual – and, in some ways, accidental – testing experiment in Vò allowed researchers to obtain a full "epidemiological picture" of the disease.
The testing began as a scrambled contingency measure after Italy's first virus-related death there on February 22nd. It is unclear why northern Italy experienced a surge in cases ahead of other parts of Europe, although Prof Crisanti and others have suggested it may be because of close commercial ties to China.
The first testing round, carried out on the town's entire population in late February, found 3 per cent of the population infected, though half of the carriers had no symptoms. After isolating all those infected, the second testing round about 10 days later showed the infection rate had dropped to 0.3 per cent.
Importantly, however, this second round identified at least six individuals who had the virus but no symptoms, meaning they could be quarantined. "If they hadn't been identified, the infection would have resumed," explained Prof Crisanti.
Rather than extensive testing, countries including the UK and the Netherlands had been pursuing a "herd immunity" strategy, which involved allowing more people to catch the virus and develop immunity to it, as a way of building up resistance in the community, though these policies are changing as the crisis intensifies.
A WHO doctor said on Tuesday that aiming for herd immunity may not be the right way to contain the virus. Dorit Nitzan, WHO health emergencies co-ordinator for Europe, said that because so little was known about the coronavirus it was "not the right time for us to recommend" a herd immunity approach. "It is a new virus and we have to learn," she said.
Britain has not been testing people with mild symptoms but is focusing on patients with serious respiratory illness. However, an abrupt reversal of policy by Boris Johnson's government this week will see the UK increase testing as part of its efforts to stop new infections and "flatten the curve" so as not to overburden health services and to reduce fatalities.
In Veneto, the authorities plan to expand the aggressive testing regime in the whole region and are seeking to take at least 11,000 swabs a day. "If someone calls the [virus] hotline and says they are sick, then everyone in the family, all their friends and the whole building get tested," said Prof Crisanti.
Luca Zaia, president of Veneto who initiated the Vò project, on Tuesday reinforced the position that widespread testing was vital. "An asymptomatic carrier can infect 10 people," he told reporters. "Swabs can save lives." – Copyright The Financial Times Limited 2020
Imperial College LondonUniversity Of PaduaWorld Health OrganizationCoronavirusAndrea CrisantiDorit NitzanLuca ZaiaTedros Adhanom Ghebreyesus
Muqtada al-Sadr: Part popular mystical figure and part militia boss who whips up a mob at a whim
UK faces severe fines if EU rules are breached in Northern Ireland, ECJ opinion suggests
Seven Israelis killed leaving synagogue in East Jerusalem
Palestinian gunman killed after shooting dead at least seven people outside Jerusalem synagogue