text:
A systematic evaluation of high-dimensional, ensemble-based regression for exploring large model spaces in microbiome analyses
Jyoti Shankar1,
Sebastian Szpakowski1,
Norma V Solis2,
Stephanie Mounaud1,
Hong Liu2,
Liliana Losada1,
William C Nierman1 &
Scott G Filler2,3
BMC Bioinformatics volume 16, Article number: 31 (2015) Cite this article
Microbiome studies incorporate next-generation sequencing to obtain profiles of microbial communities. Data generated from these experiments are high-dimensional with a rich correlation structure but modest sample sizes. A statistical model that utilizes these microbiome profiles to explain a clinical or biological endpoint needs to tackle high-dimensionality resulting from the very large space of variable configurations. Ensemble models are a class of approaches that can address high-dimensionality by aggregating information across large model spaces. Although such models are popular in fields as diverse as economics and genetics, their performance on microbiome data has been largely unexplored.
We developed a simulation framework that accurately captures the constraints of experimental microbiome data. Using this setup, we systematically evaluated a selection of both frequentist and Bayesian regression modeling ensembles. These are represented by variants of stability selection in conjunction with elastic net and spike-and-slab Bayesian model averaging (BMA), respectively. BMA ensembles that explore a larger space of models relative to stability selection variants performed better and had lower variability across simulations. However, stability selection ensembles were able to match the performance of BMA in scenarios of low sparsity where several variables had large regression coefficients.
Given a microbiome dataset of interest, we present a methodology to generate simulated data that closely mimics its characteristics in a manner that enables meaningful evaluation of analytical strategies. Our evaluation demonstrates that the largest ensembles yield the strongest performance on microbiome data with modest sample sizes and high-dimensional measurements. We also demonstrate the ability of these ensembles to identify microbiome signatures that are associated with opportunistic Candida albicans colonization during antibiotic exposure. As the focus of microbiome research evolves from pilot to translational studies, we anticipate that our strategy will aid investigators in making evaluation-based decisions for selecting appropriate analytical methods.
Biological experiments that incorporate extensive next-generation sequencing (NGS) to profile microbial communities are becoming increasingly common in the field of medical microbiology and elsewhere [1-3]. In a substantial number of these studies, the primary objective is to establish associations between the microbial community and specific biological traits or endpoints of clinical relevance. Analytical methods to test these associations follow three general directions. The first explains microbial abundance profiles in terms of biological and clinical variables [4-11]. The second utilizes these microbial profiles to classify samples into biological or clinical categories (e.g., human body sites, health and disease states, subject identities, etc.) [12-15]. A third explains a clinical or biological endpoint in terms of microbial community profiles in combination with other biological covariables [16,17]. The first two directions have formed the basis for most of the analytical work in this field. In this work, we explore the third direction.
Specifically, we used a mouse model to investigate the risk factors during antibiotic therapy for opportunistic, gastrointestinal (GI) colonization with Candida albicans, a commensal fungus in the human GI tract (Figure 1 and Table 1). C. albicans does not normally colonize mice unless the resident GI flora is perturbed by antibiotics. Therefore, we exposed mice to a single challenge with C. albicans 7 days after initiating antibiotic treatments that spanned a total of 21 days. Our initial observations indicated that some antibiotics induced a persistently higher level of C. albicans colonization compared to others. We sought further insights into these differential patterns of colonization by identifying the underlying antibiotic-induced perturbations in the bacterial and fungal GI microbiome, and the host immune factors. Hence, we employed a regression framework to explain the level of C. albicans colonization using microbiome variables, immune factors and experimental conditions as covariables.
Experimental design of the mouse microbiome study. Mice were divided into a control group and two treatment groups. The treatment groups received either vancomycin (van) or a combination of penicillin, streptomycin and gentamicin (PSG). After 7 days, some of the mice were administered a C. albicans challenge. Terminal ileum samples were collected on days 7 and 21 and fecal pellet samples on days 7, 9, 14 and 21. C. albicans colonization level was measured by counting the Colony forming units (CFU) following quantitative culture of the fecal pellets. Cytokine mRNA expression (C mRNA) was measured in the terminal ileum samples. Bacterial 16S and Fungal ITS were amplified from both the terminal ileum sections and the fecal pellets. These amplicons were sequenced with 454 pyrosequencing. The YAP bioinformatic workflow [54] was used to obtain genus-level taxonomic assignments for the bacterial and fungal sequences.
Table 1 Characteristics of data from the mouse microbiome study
We faced unique challenges in performing these analyses. Our studies utilized animal models with tens of samples (n). However, the total number of variables (p) arising from NGS measurements of the microbiome, host immune factors and other biological exposures was at least an order of magnitude larger than the number of samples (p>>n). In this scenario, a model estimated using univariate or non-regularized regression modeling strategies [6,8] is likely to overfit the data and yield findings that would not generalize well on data from newer studies [18]. Moreover, this model does not take into account the covariance structure of high-dimensional data, making the naïve assumption of orthogonality of covariables. In addition, such a model is only one among the 2p possible model configurations that could explain the data [19,20]. We therefore needed a modeling strategy which a) incorporates penalized regression to avoid overfitting, b) addresses multicollinearity, c) efficiently searches through a large number of model configurations prior to settling on the most likely model specification, and d) estimates the model parameters with low variability in the face of modest sample sizes.
Ensemble models are a class of approaches that elegantly satisfy all of the above requirements by merging information on the explanatory variables from a large collection of models to generate a ranked list of influential variables ordered by their importance [20]. These ensembles estimate robust models with stable solutions by utilizing the inherent variability in either the sample space through bootstrap methods [21,22] or the model space using Bayesian methods [23] and Markov chain Monte Carlo (MCMC) algorithms [24].
Random forests [25], a non-linear ensemble method, has been widely used for classification and prediction tasks in microbiome research [12,14]. However, random forests are not directly usable in a problem domain such as ours that requires the estimation of a model in a linear regression framework for a continuous clinical endpoint of interest. Investigators in the field have adopted penalized regression methods [4,5,14-16] that are able to effectively model multivariable covariances within the microbiome. Nevertheless, these methods have been employed in a non-ensemble setting which could potentially lead to instability in model findings.
Only a few research groups have incorporated ensemble linear regression modeling in a problem domain similar to ours [17]. Furthermore, there has been no assessment of how the characteristics of linear regression ensembles affect model performance on realistic microbiome data. We address this gap in the microbiome literature by a) rigorously evaluating the performance of a selection of frequentist and Bayesian ensemble regression modeling approaches, and b) introducing a simulation and evaluation framework that can be constructed to closely mimic the characteristics of any experimental microbiome dataset. Using this framework and a suite of metrics, we determine how various characteristics of ensembles influence model performance. We also demonstrate the application of these ensembles to our experimental mouse data. Results from our evaluation demonstrate that ensembles capable of exploring larger model spaces perform better with lower variability.
Simulation design
We generated data for our simulation study using the linear model,
$$ Y = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, $$
((1))
where Y is the response variable, X is the design matrix with n samples and p variables, β is the p-dimensional vector of regression coefficients and the random error \(\boldsymbol {\epsilon } \sim \mathcal {N}\left (0, \sigma ^{2} \cdot \text {I}_{n}\right)\) where I n is the identity matrix. We describe the construction of the components X, β and ε below.
Design matrices
We constructed two design matrices based on the experimental mouse gut bacterial and fungal microbiome profiles, respectively. We first eliminated the rows and columns which had zero counts across all samples; n and p refer to the total number of rows and columns that remained. In the bacterial matrix, n=30 and p=192 while in the fungal matrix, n=36 and p=47. We then constructed the design matrix X by log-transforming the relative abundances of genera, \(t^{\prime }_{\textit {ij}}\) where i∈{1,2,…n} and j∈{1,2,…,p}.
$$ \phi_{ij} = \left(\frac{t_{ij} + 1} {\sum_{j=1}^{p} t_{ij} + 1} \right), t^{\prime}_{ij} = log(\phi_{ij}), $$
where ϕ ij and t ij denote the relative abundance and the sequence count of bacterial or fungal genus, j in sample i, respectively. We added a smoothing constant of 1 to t ij to ensure non-zero proportions for all genera. The log-transformation ensures that all variables have a similar dynamic range across all the samples [26]. p denotes the total number of genera.
Since \(\sum _{j=1}^{p} \phi _{\textit {ij}} = 1\) for every i, the relative abundance of one of the genera is fully specified by the others. To address this redundancy, we selected one of the genera as a reference, and dropped it from the design matrix X. We chose the reference genus, j ref to be the one with the greatest abundance across all the samples. i.e.,
$$ j_{\text{ref}} = \mathop{\text{argmax}}\limits_{j \in \{1,2,\ldots\,p\}} \sum\limits_{i=1}^{n} t_{ij}. $$
Influential variables
Microbiome sequence data describes an entire community of organisms. It is not known whether this community consists of a large or a small subset of influential microbes. In addition, their degree of association with the response is also unknown. We addressed this uncertainty using the following strategies: a) considering a range of sparsity settings (% of variables specified as influential ∈{2%,3%,4%}), and b) sampling regression coefficients from either a bimodal distribution, β=±1 or a uniform distribution, \(\beta \in \mathcal {U}\).
We denote the set of specified influential variables as \(\mathcal {V}_{i}\). We randomly selected the indices of these variables and divided them into two halves \(\mathcal {V}_{i}^{1}\) and \(\mathcal {V}_{i}^{2}\). We used two alternative schemes for assigning regression coefficients β to the selected variables in \(\mathcal {V}_{i}^{1}\) and \(\mathcal {V}_{i}^{2}\), following the approach in Meinshausen et al. [22].
β=±1: \(\forall v \in \mathcal {V}_{i}^{1}, \beta _{v}=1; \: \forall v \in \mathcal {V}_{i}^{2}, \beta _{v}=-1\).
\(\beta \in \mathcal {U}\): \(\forall v \in \mathcal {V}_{i}^{1}, \beta _{v} \sim \mathcal {U}(0.5, 1.0); \: \forall v \in \mathcal {V}_{i}^{2}, \beta _{v} \sim \mathcal {U}(-1.0,-0.5)\).
The signal to noise ratio (SNR) is defined as [27]
$$ \text{SNR} = \frac{||\boldsymbol{X}\boldsymbol{\beta}||_{2}}{\sqrt{n} \sigma}, $$
where ||X β||2 is the l 2 norm of the predicted response. We computed the SNRs in our experimental mouse data by estimating an elastic net model [28]. The bacterial data had an SNR of 4.6, while the fungal data had an SNR of 5.2. In our simulation, we applied three different SNRs of 0.25, 4.60 and 16.00, that correspond to high, medium, and low noise settings, respectively. For a given SNR, we sampled the random error \(\epsilon _{i} \sim \mathcal {N}(0, \sigma ^{2})\) where i∈{1,2,…n} and where σ is chosen as:
$$ \sigma = \frac{||\boldsymbol{X}\boldsymbol{\beta}||_{2}}{\sqrt{n} \cdot \text{SNR}}. $$
It is easier to recover the influential variables under a higher SNR since the response has a larger association with the variables relative to the random error term.
For each combination of X, β and SNR, we performed a total of 130 simulations and report results aggregated over these runs, making our findings robust to variations. We conducted our evaluation and analysis in the R language for statistical computing [29] and have released our code at GitHub [30].
Modeling approaches
In our evaluation, elastic net with cross-validation (ENC) was the baseline non-ensemble penalized regression method. We chose the remaining frequentist and Bayesian ensemble modeling approaches with the objective of determining how the size of the ensemble, construction of the model space, and the choice of regularization parameters within the ensemble influenced performance. We summarize these approaches in Table 2 and describe them in detail below.
Table 2 Summary of the modeling approaches included in the evaluation
Elastic net with cross-validation (ENC)
The ENC procedure [28] is a specific instance of a penalized regression model [31,32].
$$ \hat{\boldsymbol{\beta}} \,=\, \mathop{\text{argmax}}\limits_{\boldsymbol{\beta}} \!\left\{\! \sum\limits_{i=1}^{n} |y_{i} \,-\, X_{i}\boldsymbol{\beta}|^{2} \,+\, \lambda\!\left(\!\alpha ||\boldsymbol{\beta}||_{1} \,+\, (1-\alpha)\!\frac{1}{2}\!||\boldsymbol{\beta}||^{2}_{2}\!\right) \!\right\}\!, $$
where \(\hat {\boldsymbol {\beta }}\) is the solution to the estimation problem, the penalty terms, \(\|\boldsymbol {\beta }\|_{1} = \sum _{i=1}^{p} |\boldsymbol {\beta }_{i}|\) and \(\|\boldsymbol {\beta }\|_{2} = \sqrt {\sum _{i=1}^{p} |\boldsymbol {\beta }_{i}|^{2}}\) are the l 1 and l 2 norms of the coefficient vector β, \(\sum _{i=1}^{n} |y_{i} - X_{i}\boldsymbol {\beta }|^{2}\) is the residual sum of squares (RSS), λ is the tuning parameter that penalizes the RSS by the size of the regression coefficients and α is a tuning parameter that balances the l 1 and l 2 penalties. An α of 1 promotes sparsity in the model, while an α of 0 ensures that correlated variables are assigned similar regression coefficients. An optimal value of α finds a balance between the two penalties. We employed leave-one-out cross-validation to determine the optimal combination of α and λ (λ optimal =λ ENC ) that minimized the mean squared error (MSE) of the model [31]. We used the implementation of ENC available in the glmnet R package [32].
Frequentist ensembles
We evaluated frequentist ensembles that were variations on the ensemble strategy originally described by Meinshausen and Bühlmann (MB) [22]. In their work, MB proposed a method to improve stability of variable selection within the least absolute shrinkage and selection operator (LASSO) approach [33]. Since ENC is a generalization of the LASSO approach and our baseline, we replaced LASSO with ENC in MB's stability selection procedure.
Ensembles based on subsamples (PS, LS, SS, SSW)
Within MB's stability selection procedure, we first generated 100 subsamples of the data. From each subsample, we estimated regression coefficients β on a grid of regularization parameters λ∈Λ. Λ consisted of 100 λ values obtained using the Least angle regression (LARS) algorithm within ENC [31,34] implemented in the glmnet R package [32]. Given the set of selected variables at any λ, S(λ)={j:β j (λ)≠0}, the inclusion probability P(j∈S(λ)) of any variable j∈{1,2,…p} was computed as
$$ P(j \in S(\lambda))= \frac{1}{B} \sum\limits_{b=1}^{B} 1_{j \in S_{\lambda}(b)}, $$
where b is the index of any given subsample, B=100 is the total number of subsamples and S λ (b) is the set of selected variables at regularization parameter λ in subsample b. To minimize the expected number of false positives within the selected set of variables, S(λ), MB developed an algorithm to select both the optimal regularization parameter λ optimal =λ MB and a stable set of variables, \(\hat {S}^{stable} \subseteq S(\lambda _{\textit {MB}})\) [22]. We adopted an implementation of MB's algorithm from the quadrupen R package [35]. Since MB's algorithm was designed to reduce false positives and employed subsampling to construct the model space, we called this modeling approach, PS. In addition to PS, we evaluated ensemble variants which enabled us to determine how performance was influenced by the choice of the regularization parameter λ optimal , size and nature of the model space, and the computation strategy for inclusion probabilities.
In the first variant of PS, we chose the single regularization parameter λ optimal =λ ENC , specified by the ENC procedure in glmnet [32]. Unlike PS that performed variable selection at λ MB , we computed the inclusion probabilities of the p variables across the 100 subsamples at λ ENC using Equation 7. We termed this variant LS since it operated at a single λ optimal and employed subsampling to construct the model space.
In the next variant, we expanded the size of the model space by computing the average inclusion probabilities, I j , of the variables across the 100 regularization parameters λ∈Λ,
$$ I_{j} = \frac{1}{100} \cdot \sum\limits_{i = 1}^{100} P(j \in S(\lambda_{i})), $$
where j∈{1,2,…p}. The approach assigned high inclusion probabilities to the stable variables which were most consistently selected across the Λ grid and employed subsampling to construct the model space. Hence we termed this variant SS.
We also considered an alternative definition of inclusion probability, \({I}^{w}_{j}, j \in \{1,2,\ldots,p\}\) based on a suggestion from our referee.
$$ {I^{w}_{j}} = \sum\limits_{i = 1}^{100} w_{\lambda_{i}} \cdot P(j \in S(\lambda_{i})), $$
where \(w_{\lambda _{i}}\), the weight assigned to λ i , is proportional to the average number of non-zero variables at λ i
$$ w_{\lambda_{i}} = \frac{{\sum\nolimits}_{b=1}^{B} 1_{j \in S_{\lambda_{i}}(b)}}{ {\sum\nolimits}_{i=1}^{100} {\sum\nolimits}_{b=1}^{B} 1_{j \in S_{\lambda_{i}}(b)}}, $$
((10))
where b is the index of the subsamples and B = 100 is the total number of subsamples. We termed this variant SSW.
Ensembles based on resamples (PR, LR, SR, SRW)
Since MB [22,27] recommend the use of resampling as an equivalent alternative to subsampling in the context of stability selection, we varied the nature of model space construction within the frequentist ensembles by adopting bootstrap resampling as an alternative to subsampling with size \(\lfloor \frac {n}{2}\rfloor \). Subsampling without replacement and sampling at random with replacement (bootstrap resampling) are examples of schemes within a family of exchangeably weighted bootstrap schemes [36,37]. The equivalence of the two bootstrapping schemes has been described in [38] and [39] and demonstrated through theoretical results and extensive simulations of non-linear regression [40]. In addition, the two schemes have very similar statistical properties [41]. It has also been independently shown that bootstrap resampling improves stability of the LASSO procedure, which is a special case of ENC [42].
We implemented the resampling alternatives to PS, LS, SS and SSW, and termed them PR, LR, SR and SRW, respectively.
Bayesian ensembles
In addition to the frequentist ensembles, we evaluated Bayesian ensembles that further expanded the size of the model space explored. These larger ensembles were based on spike-and-slab Bayesian model averaging (BMA) [24] implemented in the BoomSpikeSlab R package [43]. Given a set of multicollinear variables, spike-and-slab BMA assigns higher inclusion probabilities to influential variables highly associated with the response. Strongly correlated variables that are equally associated with the response are assigned similar inclusion probabilities. Thus the BMA framework is conceptually similar to our baseline frequentist penalized regression approach, ENC, and its ensemble variants.
The spike-and-slab prior consists of two components and is described in detail in [44,45]. The spike component of the prior is modeled by a Bernoulli distribution for each variable and is described by
$$ {\boldsymbol{\gamma}} \sim \prod\limits_{i=1}^{p} \pi_{i}^{(\gamma_{i})} (1- \pi_{i})^{(1-\gamma_{i})}, $$
where γ is a vector of binary indicators, γ i ;i∈{1,2…p}. If γ i =1, the i th variable is selected as influential. If γ i =0, the i th variable is not influential and β i =0. π i is the probability of the i th variable being influential and computed as \(\pi _{i}=\frac {k}{p}\). k is the expected model size and specifies the number of variable coefficients expected to be non-zero.
Conditional on the set of influential variables {i:γ i =1}, the slab component of the prior distribution models the regression coefficients β for these variables. The distribution is a variant of the Zellner's g-prior [46] and is given by
$$ {\boldsymbol \beta} | \left(\sigma^{2}, {\boldsymbol \gamma} \right) \sim \mathcal{N} \left({\boldsymbol b}, \boldsymbol{F_{\gamma}}^{-1}\right), $$
where b denotes the prior expectation of β and is set to 0 in our experiments. F γ is the sub-matrix that corresponds to the influential variables {i:γ i =1} within the full prior Fisher information matrix,
$$ \boldsymbol{F} = \frac{g}{n} \cdot \frac{\boldsymbol{X^{T} X}}{\sigma^{2}}. $$
In the event of multicollinearity, F may not always be positive definite. Therefore, the BMA implementation within BoomSpikeSlab ensures a proper posterior distribution by linearly interpolating X T X with its diagonal to obtain the smoothed Fisher information matrix [44,45],
$$ \boldsymbol{F}_{smooth} = \frac{g}{n} \cdot \frac{w \times \boldsymbol{X^{T} X} + (1-w) \times \text{diag}(\boldsymbol{X^{T} X})}{\sigma^{2}}, $$
where g is set to 1 and w=0.5. σ 2 is the variance of the random error ε in the regression model and distributed as
$$ \frac{1}{\sigma^{2}}| \boldsymbol{\gamma} \sim \Gamma\left(\frac{\nu}{2}, \frac{ss}{2}\right), $$
where \(\Gamma \left (\frac {\nu }{2}, \frac {ss}{2}\right)\) represents a gamma distribution with mean \(\frac {\nu }{ss}\) and variance \(\frac {\nu }{ss^{2}}\). We retained the default values of ν=0.01 and \(ss = 0.5 \cdot {s}_{y}^{2}\) from the BoomSpikeSlab R package [43]. \({s}_{y}^{2}\) is the standard deviation of the response. We set the expected model size, k, to the default value of 1 in the variant termed BMA. For the variant termed BMAC, we estimated k with five-fold cross-validation to minimize the MSE. This enabled the BMAC variant to adapt to the inherent sparsity setting in the data.
The BMA procedure incorporates a MCMC algorithm that traverses a very large space of models to estimate the posterior distribution of the regression model parameters [44]. For our datasets, the running means of the selection indicators and the regression coefficients converged well before 10,000 MCMC iterations. Hence, we ran the MCMC algorithm for 10,000 iterations and discarded the initial 1000 iterations as burn-in. We then estimated the variable inclusion probabilities from the remaining 9000 iterations. The inclusion probability for the i th variable is the proportion of iterations (or draws) with non-zero regression coefficients
$$ P(\gamma_{i}=1) = \frac{1}{D} \sum\limits_{d=1}^{D} 1_{\beta_{i} \ne 0}(M_{d}), $$
where D=9000 is the total number of draws of model parameters from the posterior distribution, and M d indicates the model parameters at iteration d.
Variable selection
In high-dimensional settings, variable selection results in a sparse set of variables which provide the best explanation for the data. ENC performs variable selection by assigning the non-influential regression coefficients to zero. PS and PR directly compute the set of influential variables. For all the other approaches that generate variable inclusion probabilities, we applied a variable selection algorithm. In this algorithm, we first ranked the inclusion probabilities in decreasing order. We then computed the first order lagged differences between the ranked inclusion probabilities, identified the largest gap in this sequence, and selected the variables above this gap as influential. Suppose IP i ,i∈{1,2,…p} denote the ranked inclusion probabilities, then the index that corresponds to the largest gap in inclusion probabilities is
$$ \hat{i} = \mathop{\text{argmax}}\limits_{i}\left\{\text{IP}_{i} - \text{IP}_{i+1} \right\}, $$
and the set \(\hat {\mathcal {V}}_{i} = \{v_{1}, v_{2}, \ldots v_{\hat {i}}\}\) is the chosen set of influential variables. We refer to this strategy as the lagged differences (LD) algorithm.
Evaluation metrics
We evaluated the approaches on the basis of their ability to select the truly influential variables as well as their ability to rank these variables accurately.
ROC curves and AUC score The receiver operating characteristic (ROC) curve is a visual depiction of the variable selection performance of an approach across a range of thresholds on the inclusion probability. We constructed the ROC curve for each approach by plotting true positive rate (TPR) versus false positive rate (FPR) while sweeping across a sequence of 22 inclusion probability thresholds between 0.00 and 1.01 on the logarithmic scale. True positives (TP) are those variables which were correctly identified as influential by an approach. False positives (FP) are those which were incorrectly identified as influential by an approach. TPR and FPR are defined as follows
$$ TPR = \frac{TP} {|\mathcal{V}_{i}|} \,\, FPR = \frac{FP}{p-|\mathcal{V}_{i}|}, $$
where p is the total number of variables and \(\mathcal {V}_{i}\) is the set of all influential variables in the simulated data. In constructing the ROC curve for any given approach, variables with inclusion probabilities above a given threshold were counted as influential variables selected at that threshold. We also computed the area under the ROC curve (AUC) score for each of the approaches. The AUC is a single metric that quantifies the performance of any given approach across the complete range of inclusion probability thresholds [47]. Both the ROC curve and the AUC metric allowed us to assess the performance of approaches that generated inclusion probabilities for variables but did not perform variable selection. Therefore, these metrics evaluated model performance independent of any particular algorithm for variable selection.
F-score
The F-score is a single metric that measures the accuracy of variable selection with respect to the truly influential variables. It is the harmonic mean of precision and recall and is given by
$$ \begin{aligned} \text{Precision} &= \frac{\text{TP}}{\text{TP+FP}}\\ \text{Recall} &= \frac{TP} {|\mathcal{V}_{i}|}\\ \text{F-score} &= 2 \left(\frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \right). \end{aligned} $$
Variable ranking
We measured the Spearman's rank correlation coefficient between the simulation-assigned ranks r=(r 1,r 2,…r p ) and the observed ranks \(\hat {r} = (\hat {r}_{1}, \hat {r}_{2}, \ldots \hat {r}_{p})\) of the variables. For variable i, r i was based on the absolute value of its regression coefficient while \(\hat {r}_{i}\) was based on its inclusion probability from a given approach.
The Spearman's rank correlation between r and \(\hat {r}\) is specified by
$$ \rho = \frac{\sum_{i=1}^{p} (r_{i} - \bar{r}) (\hat{r}_{i} - \bar{\hat{r}})} { \sqrt {\sum_{i=1}^{p} (r_{i} - \bar{r})^{2} \sum_{i=1}^{p} (\hat{r}_{i} - \bar{\hat{r}})^{2} }}, $$
where \(\bar {r}\) and \(\bar {\hat {r}}\) are the mean ranks of the true regression coefficients and the inclusion probabilities, respectively.
Receiver operating characteristic (ROC)
Figure 2 shows the mean ROC curves, computed over 130 simulations, for approaches that assign inclusion probabilities to variables without performing variable selection. Good performance is characterized by high TPR and low FPR over a wide range of inclusion probability thresholds. The diagonal line on the plots represents random variable selection performance. ROC curves below the diagonal indicate poor performance while those above have better than random performance.
Receiver operating characteristic (ROC) curves. Mean ROC curves across 130 simulations are shown for approaches that do not perform variable selection. An ideal ROC has a high TPR and a low FPR over a wide range of thresholds. S denotes the sparsity setting or the proportion of influential variables in the data. The diagonal dashed line represents random variable selection performance. a. β=±1 and b. \({\beta \in \mathcal {U}}\).
Overall, BMA variants consistently outperformed all the other approaches. Performance of all approaches deteriorated in high dimensions and with increasing number of influential variables, demonstrating that all ensembles had uniform difficulty in variable selection within less-sparse settings. Indeed, the setting with the highest percentage (4%) of influential variables and largest regression coefficients (β=±1) presented the most challenging scenario. The performance gap between BMA and the other approaches decreased in this case.
The relatively strong performance of all approaches in the low-dimensional fungal setting was most likely due to the smaller space of variable configurations. In most scenarios, SS and SR performed better than LS and LR approaches, illustrating the advantage of moving from a single λ optimal to a larger model space consisting of a grid λ∈Λ.
At higher SNRs, BMA with cross-validated expected model size (BMAC) performed better than the other approaches. We also note that higher SNRs were uniformly associated with better performance.
Area under the ROC curve (AUC)
Figure 3 shows the variability in AUC across 130 simulations for the approaches that assign inclusion probabilities. The corresponding median AUCs are shown in Additional file 1. An AUC of 1.0 is ideal and corresponds to a scenario where an approach consistently ranks influential variables higher than non-influential variables. In our simulations, the trends in AUC mirrored the patterns in the ROC curves. BMA variants and specifically, BMAC, obtained the highest AUC along with the lowest variability, showing the advantages of exploring the largest model space among all approaches.
Variation in area under the ROC curve (AUC) across 130 simulations are shown for approaches that do not perform variable selection. Points beyond the end of the whiskers denote outliers. An AUC of 1.0 is ideal. S denotes the sparsity setting or the proportion of influential variables in the data. a. β=±1 and b. \({\beta \in \mathcal {U}}\).
While BMA variants had a near perfect AUC in the low-dimensional fungal setting, the AUCs for non-BMA approaches were also higher in this setting. Although SS and SR had AUCs similar to BMA in the fungal setting, they had a much higher variability. The resampling variants, LR and SR, yielded small but consistent improvements in median AUC relative to their subsampling counterparts in the high-dimensional setting.
The lowest SNRs resulted in the lowest AUCs and the highest variability. Since SNRs of <0.25 are not representative of our experimental datasets and in addition, yielded uniformly poor performance across all ensembles, we report results only for medium and high SNR situations in the evaluations that follow.
Higher values of the F-score indicate a good balance between precision and recall. Figure 4 shows the variation in F-scores across 130 simulation runs for all approaches while Additional file 1 shows the medians. For methods that did not perform variable selection, we applied the inclusion probability threshold from the LD algorithm to select the top influential variables.
Variation in F-score across 130 simulations. Points beyond the end of the whiskers denote outliers. An F-score of 1.0 is ideal. The LD algorithm was used to select variables for approaches that do not perform variable selection. SNR=0.25 is not shown. S denotes the sparsity setting or the proportion of influential variables in the data. a. β=±1 and b. \({\beta \in \mathcal {U}}\).
BMAC outperformed all the other approaches across all settings. BMA was a close second, however, its performance deteriorated at lower sparsity settings. As was pointed out by one of our referees, this finding shows the benefits of cross-validating the expected model size within BMAC, enabling it to adapt better to the sparsity in the data. The PR algorithm performed as well as BMAC in the low-sparsity and medium SNR settings within high dimensions and in the presence of large regression coefficients (β=±1). This would suggest that resampling conferred a distinct advantage over subsampling, enabling PR to perform well despite its small model space.
In lower dimensions, BMA approaches, with the largest ensembles, outperformed all other approaches across all SNR settings. SS and SR performed similar to BMA in a number of low-dimensional fungal settings but the variability in their performance was substantially higher.
Spearman's rank correlation
Figure 5 shows the variable ranking performance measured using Spearman's rank correlation between the estimated and true variable ranks for each of the approaches for \(\beta \in \mathcal {U}\). The corresponding median correlations are shown in Additional file 1. Overall, the rank correlation decreased with an increase in dimensionality. BMA approaches showed higher correlation and lower variability relative to the other approaches in high dimensions. However, all approaches performed similarly in low dimensions.
Variation in Spearman's rank correlation ( \(\beta \in \boldsymbol{\mathcal {U}}\) ) across 130 simulations are shown for approaches that do not perform variable selection. Points beyond the end of the whiskers denote outliers. A correlation of 1.0 is ideal. Higher correlation indicates that the approach is able to capture the original variable rank more accurately. SNR=0.25 is not shown. S denotes the sparsity setting or the proportion of influential variables in the data.
Additional simulations
We performed two additional simulations to address points raised by our referees.
Performance without log-transformation
In this simulation, we examined the performance of all ensembles on relative abundance data that were not log-transformed. Data without log-transformation showed similar trends in performance as log-transformed data (Additional file 2). However, the number of outliers increased across all metrics in the data without log-transformation. This suggests that log-transformation of relative abundances decreases variability in the performance of ensembles.
Weighted average of inclusion probabilities
We also performed simulations to compare SS and SR with their weighted counterparts, SSW and SRW, computed using Equation 9. Overall, the performance of the weighted approach was not consistently different from the unweighted approach (Additional file 3). However, there were minor increases in the median performance of the weighted approach in high-SNR settings.
Application to mouse microbiome data
We now illustrate the application of the ensembles (Table 2) to identify influential variables in our experimental mouse microbiome data collected from the terminal ileum. Briefly, the gastrointestinal tract is an ecological niche for both bacterial and fungal flora. By depleting the competing bacterial microbiota, antibiotics create vacancies in the niche that co-existing fungi can repopulate. However, a high level of colonization by a single fungal species such as C. albicans is an unfavorable clinical outcome [48], especially in immunosuppressed and dysbiotic patient populations [49-51]. We therefore designed our mouse microbiome study to examine the factors influencing C. albicans colonization of the gastrointestinal tract following initiation of antibiotic treatment.
The experimental design of our mouse microbiome study is summarized in Figure 1 and Table 1. Following exposure to antibiotics and C. albicans, we profiled the microbial communities in the mouse gut by amplifying and sequencing the taxonomically discriminant bacterial 16S rDNA [52] from the V3-V5 variable region and the fungal ITS rDNA [53]. Using the YAP bioinformatic workflow [54], we obtained taxonomic assignments that reach genus-level resolution for both bacteria and fungi. Assignments unclassified at the genus-level were annotated with the prefix UC. The median number of reads per sample was 3500 for bacterial 16S and 2000 for fungal ITS regions. We built two linear regression models to assess the association of bacteria and fungi with the level of C. albicans colonization. We estimated the bacterial models from 30 samples and the fungal models from 36 samples. Both the bacterial and fungal models were specified as follows
$$ Y = {\boldsymbol X}_{g}\boldsymbol{\beta}_{g} + {\boldsymbol X}_{c}\boldsymbol{\beta}_{c} + {\boldsymbol X}_{a}\boldsymbol{\beta}_{a} + \boldsymbol{\epsilon}, $$
where Y is the colonization level measured in \(log \left (\frac {\text {Colony forming units (CFU)}}{\text {grams of fecal matter}}\right)\), X g is a vector of log relative abundances of sequences assigned to bacterial or fungal genera, X c is a vector of log normalized mRNA expression levels of cytokines, X a is a vector of 5 binary variables indicating antibiotic treatment (vancomycin or PSG: penicillin, streptomycin and gentamicin) for a span of 7 or 21 days and exposure to C. albicans on the 7th day, and ε is the residual error in the model.
Since the microbial proportions sum to one, the proportion of any given genus is known given the proportions of all the other genera. To avoid this redundancy, we could exclude the proportion of any one reference genus. However, doing so would mean we are unable to obtain inclusion probability for this genus. We thus selected two reference genera with the highest abundances across all samples and repeated the model building procedure twice, each time excluding one of the two reference genera from X g . We estimated the final set of selected variables and inclusion probabilities for all genera by averaging across these two models.
We applied each of the approaches (Table 2) from the simulation study to this data to discover the microbiome-immune-antibiotic signatures associated with the level of C. albicans colonization. The inclusion probability for any variable reflects its importance to the response. For methods that do not perform variable selection, we used the LD algorithm to derive an inclusion probability threshold for identifying the top influential variables. Since ENC, PS and PR directly provide us with lists of influential variables and do not assign inclusion probabilities, we annotated the inclusion probabilities for these influential variables as 100%.
Mouse microbiome findings
Figure 6 shows the bacteria and cytokines identified as influential by the approaches along with their inclusion probabilities (in %). ENC showed the least sparsity and selected the maximum number of bacteria and host immune cytokines. Although Veillonella was identified as an influential variable in all approaches, only BMA, BMAC, SS and SR assigned it the highest inclusion probability. However, unlike SS and SR, the difference in the inclusion probabilities of Veillonella and the other influential variables was much larger in the BMA models.
Candida colonizatin: Influential variables in the bacterial model. Variables selected in each of the modeling approaches are highlighted in green. The LD algorithm was utilized for selecting variables for approaches that do not perform variable selection. Values in the cells are inclusion probabilities expressed in %. ENC, PS and PR do not generate inclusion probabilities. In these models, an inclusion probability of 100% indicates that the variable was selected as influential.
Figure 7 shows results from the fungal model. Treatment with PSG for a span of (a) 7 days, and (b) 21 days along with exposure to C. albicans were both assigned very high inclusion probabilities by BMA and BMAC. Unlike the bacterial model, the highest-ranking variables in SS and SR were different from those in BMA approaches. PR and PS shared common variables with both SS and SR and the BMA approaches. The other single λ approaches (LR, LS) showed some similarities with SS and SR, however, their inclusion probability assignments were substantially different. ENC showed a lack of sparsity analogous to the bacterial models.
Candida colonization: Influential variables in the fungal model. Variables selected in each of the modeling approaches are highlighted in green. The LD algorithm was utilized for selecting variables for approaches that do not perform variable selection. Values in the cells are inclusion probabilities expressed in %. ENC, PS and PR do not generate inclusion probabilities. In these models, an inclusion probability of 100% indicates that the variable was selected as influential.
Viewed together, findings from the bacterial and fungal models indicate that the type of antibiotic, span of exposure to the antibiotic and specific antibiotic-induced changes in the bacterial microbiota substantially influenced the levels of C. albicans colonization. The concurrent effects of the host immune cytokines and the co-existing fungal microbiota on the level of C. albicans colonization were weaker. These findings are promising since they suggest that tailoring antibiotic regimens as well as concomitant modulation of the microbiota during antibiotic administration could be employed to reduce opportunistic C. albicans colonization.
Comparison of approaches
To further examine similarities and differences in the workings of each of the modeling approaches, we plotted the inclusion probability versus rank for all variables across the ensembles (Figure 8). In both high and low-dimensional settings, the approaches separated out into three distinct groups. In the first group that included BMA and BMAC, inclusion probabilities showed the steepest decay. In the second, comprising the SS and SR, the decay was more gradual. The single λ approaches constituted the third group. Here the rate of decay was similar to SS and SR, however the dynamic range of inclusion probabilities was larger. The sharper decay in the BMA methods indicates a more discriminative ranking that allows a distinct separation of the influential variables from the non-influential ones.
Comparison of inclusion probability profiles across all approaches. The top panels show the sorted inclusion probabilities plotted against the rank of the variables in the model. The bottom panels show the pairwise similarities between the inclusion probability profiles of all the approaches. The similarity between any pair of approaches i and j is given as \(S_{\textit {ij}}=\frac {1}{1+D_{\textit {ij}}}\), where D ij is the Euclidean distance between the inclusion probability vectors of approaches i and j. We generated the dendrogram using agglomerative clustering with the centroid linkage method. We then ordered the correlogram using the order of models generated in the dendrogram. a. Bacterial models, b. Fungal models.
On clustering the approaches based on Euclidean distance between the inclusion probability vectors, the single λ approaches were further away from PS, PR, SS, SR and the BMA approaches, indicating their divergent performance. The proximity of SS and SR to BMA approaches suggests that SS and SR, computed over a much larger number of subsamples or resamples, could potentially approach the performance profile of BMA.
Microbiome studies constitute a recently developed area of research that incorporate NGS to profile communities of diverse microbes residing in a variety of ecosystems. As a result, microbiome data shares many of the characteristics of NGS data that make it challenging to analyze. These challenges arise from high-dimensionality, limited sample sizes, multicollinearity within covariables, modest SNRs and a very large space of models. Ensemble modeling is able to provide a comprehensive solution to these analytical challenges. However, except for non-linear ensembles such as random forests, ensemble modeling has not been adopted widely in microbiome research.
Our goal in this paper was to familiarize investigators in the field with the characteristics of both frequentist and Bayesian ensemble-based regression approaches and present an evaluation of these approaches on realistic microbiome data. We also introduce a simulation framework that can be built from any experimental microbiome dataset to closely mimic the characteristics of real data. We demonstrate the usefulness of this framework in evaluating analytical approaches using a suite of metrics that assess various facets of modeling performance. Unlike other simulation strategies [4,16] that sample covariates from parametric distributions, we directly utilize the covariance structure, SNR and the dimensions inherent in our experimental mouse data for our simulation. Even though parametric simulation approaches enable generation of arbitrary number of samples, we expect the findings from our simulation-based evaluation to translate better to experimental datasets. Our implementation, in the R statistical language, is freely available from GitHub [30], making it convenient for researchers in the microbiome research community to adopt and apply our evaluation methodology to their own datasets without the need for expensive ground truth annotations.
We found that the Bayesian approaches had several favorable attributes that distinguished them from the other ensembles we evaluated. In realistic settings with medium SNRs and uniformly distributed coefficients for influential variables (\(\beta \in \mathcal {U}\)), they performed consistently better or at par with frequentist approaches depending on the number of influential variables. Furthermore, they showed substantially less variation across replications. Selecting the optimal expected model size via cross-validation (BMAC) improved performance in more challenging high-dimensional settings with a large number of influential variables. PR, the resampling variant of the original stability selection algorithm [22] performed as well as BMAC in these scenarios, highlighting its ability to perform effective variable selection when there are several variables with large regression coefficients. On the whole, the resampling variants had small but consistent performance gains over their subsampling counterparts, suggesting that resampling was able to create an improved model space relative to subsampling. Even though SS, SR, PS and PR, the frequentist ensembles, performed reasonably well at lower dimensions, they exhibited substantially more variability in their performance. Our findings thus enabled us to pick BMAC for our experimental setting with modest sample sizes and high-dimensionality in measurements.
We note that BMAC uses the data for both estimating a hyperparameter of the prior distribution and for computing the posterior distribution. It is therefore not a full Bayesian approach. An ideal alternative would be to employ a full Bayesian model that includes a hyperprior for the prior inclusion probabilities [55]. While our work has systematically evaluated a powerful set of sparse linear regression modeling methods that average across large model spaces in the context of continuous response variables, a future direction would be to extend these evaluations for analyzing categorical responses and time-to-event outcomes that are prevalent in clinical settings. These models include logistic, multinomial, and survival regression.
Our analysis could have been performed at the higher taxonomic levels of phylum, class, order and family. However, we selected the genus-level primarily because it provided the closest taxonomic resolution to our response (C. albicans colonization). A limitation of our approach, which is shared by other analytical approaches such as [4,16,17], is the assumption that taxonomic assignment for all organisms at all phylogenetic levels is equally accurate. As a result, covariables are treated alike regardless of classification accuracy. However in practice, classification error could vary across phylogenetic branches as well as taxonomic levels and depends on factors such as the quality of annotated taxonomic databases as well as the other parameters of underlying bioinformatic algorithms [56]. A crucial avenue of future research would be to integrate information from models constructed from taxonomic assignments at varying classification accuracies.
Another area that we have not explored in this work is the inclusion of interactions and variable sub-structures within the high-dimensional regression framework. In the microbiome setting, this would involve several higher-order interaction terms among taxa. Many such symbiotic and antagonistic relationships are known to exist among taxa and cytokines but are hard to assess in analytical settings. Including these interactions is computationally challenging because they would increase the model space exponentially. Recent developments in statistical algorithms that efficiently explore this additional complex model space [57,58] hold promise for making this problem more tractable at the scale of microbiome data. In the context of biological interpretations, it could be useful to integrate variable clustering with ensemble-based regression in a framework similar to the one proposed in [59] to obtain a greater understanding of the most relevant dynamics and relationships in the community.
In conclusion, as microbiome studies evolve towards translational settings, analysts are likely to face challenges in selecting appropriate modeling strategies that yield consistent and stable performance with low variability [60]. Therefore, we expect that our research will provide both insights for choosing among ensemble methods and an evaluation framework critical for making an objective selection.
Area under the ROC curve
BMA:
Bayesian model averaging
BMAC:
Bayesian model averaging with cross-validation for expected model size
Bolasso :
Bootstrap-enhanced LASSO
CFU:
Colony forming units
Cross-validation
Expected model size
ENC:
Elastic net with cross-validation
FPR:
False positive rate
Glyceraldehyde-3-Phosphate Dehydrogenase
GI:
ITS:
Internal transcribed spacer
LARS:
Least angle regression
LASSO:
Least absolute shrinkage and selection operator
Lagged differences
Stability selection on resamples using λ ENC
LS:
Stability selection on subsamples using λ ENC
Meinshausen &
Bühlmann; MCMC:
Markov chain Monte Carlo
MSE:
Mean squared error
NIH:
Stability selection on resamples minimizing expected number of false positives
Stability selection on subsamples minimizing expected number of false positives
PSG:
penicillin, streptomycin, gentamicin
ROC:
Receiver operating characteristic
Residual sum of squares
SNR:
SR:
Stability selection on resamples using unweighted average inclusion probabilities across λ∈Λ
SRW:
Stability selection on resamples using weighted average inclusion probabilities across λ∈Λ
Stability selection on subsamples using unweighted average inclusion probabilities across λ∈Λ
SSW:
Stability selection on subsamples using weighted average inclusion probabilities across λ∈Λ
TP:
True positives
TPR:
True positive rate
UC:
The NIH HMP Working Group, Peterson J, Garges S, Giovanni M, McInnes P, Wang L, et al.The NIH human microbiome project. Genome Res. 2009; 19(12):2317–23. doi:10.1101/gr.096651.109.
Article CAS PubMed Central Google Scholar
Hood L. Tackling the microbiome. Science. 2012; 336(6086):1209. doi:10.1126/science.1225475.
Gilbert JA, Bailey M, Field D, Fierer N, Fuhrman JA, Hu B, et al.The Earth Microbiome Project: The Meeting Report for the 1st International Earth Microbiome Project Conference, Shenzhen, China, June 13th-15th 2011. Standards Genomic Sci. 2011; 5(2):243–7. doi:10.4056/sigs.2134923.
Chen J, Li H. Variable selection for sparse dirichlet-multinomial regression with an application to microbiome data analysis. Ann Appl Stat. 2013; 7(1):418–42. doi:10.1214/12-aoas592.
Xia F, Chen J, Fung WK, Li H. A logistic normal multinomial regression model for microbiome compositional data analysis. Biometrics. 2013; 69(4):1053–63. doi:10.1111/biom.12079.
White JR, Nagarajan N, Pop M. Statistical methods for detecting differentially abundant features in clinical metagenomic samples. PLoS Comput Biol. 2009; 5(4):1000352. doi:10.1371/journal.pcbi.1000352.
Paulson JN, Stine OC, Bravo HC, Pop M. Differential abundance analysis for microbial marker-gene surveys. Nat Methods. 2013; 10(12):1200–2. doi:10.1038/nmeth.2658.
Segata N, Izard J, Waldron L, Gevers D, Miropolsky L, Garrett W, et al.Metagenomic biomarker discovery and explanation. Genome Biol. 2011; 12(6):60. doi:10.1186/gb-2011-12-6-r60.
Taur Y, Xavier JB, Lipuma L, Ubeda C, Goldberg J, Gobourne A, et al.Intestinal domination and the risk of bacteremia in patients undergoing allogeneic hematopoietic stem cell transplantation. Clin Infectious Diseases. 2012; 55(7):905–14. doi:10.1093/cid/cis580.
Lozupone C, Knight R. UniFrac: a new phylogenetic method for comparing microbial communities. Appl Environ Microbiol. 2005; 71(12):8228–35. doi:10.1128/aem.71.12.8228-8235.2005.
Chen J, Bittinger K, Charlson ES, Hoffmann C, Lewis J, Wu GD, et al.Associating microbiome composition with environmental covariates using generalized UniFrac distances. Bioinformatics. 2012; 28(16):2106–13. doi:10.1093/bioinformatics/bts342.
Beck D, Foster JA. Machine learning techniques accurately classify microbial communities by bacterial vaginosis characteristics. PLoS One. 2014; 9(2):87830. doi:10.1371/journal.pone.0087830.
Knights D, Costello EK, Knight R. Supervised classification of human microbiota. FEMS Microbiol Rev. 2011; 35(2):343–59. doi:10.1111/j.1574-6976.2010.00251.x.
Statnikov A, Henaff M, Narendra V, Konganti K, Li Z, Yang L, et al.A comprehensive evaluation of multicategory classification methods for microbiomic data. Microbiome. 2013; 1(11). doi:10.1186/2049-2618-1-11.
Waldron L, Pintilie M, Tsao M-SS, Shepherd FA, Huttenhower C, Jurisica I. Optimized application of penalized regression methods to diverse genomic data. Bioinformatics. 2011; 27(24):3399–406. doi:10.1093/bioinformatics/btr591.
Garcia TP, Müller S, Carroll RJ, Walzem RL. Identification of important regressor groups, subgroups and individuals via regularization methods: application to gut microbiome data. Bioinformatics. 2014; 30(6):831–7. doi:10.1093/bioinformatics/btt608.
Lin W, Shi P, Feng R, Li H. Variable selection in regression with compositional covariates. Biometrika. 2014; 101(4):785–797. doi:10.1093/biomet/asu031.
Johnstone IM, Titterington DM. Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2009; 367(1906):4237–53. doi:10.1098/rsta.2009.0159.
James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning: with Applications in R, 1st edn. New York, NY, USA: Springer Series in Statistics, Springer; 2013. doi:10.1007/978-1-4614-7138-7.
Breiman L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Stat Sci. 2001; 16(3):199–231. doi:10.1214/ss/1009213726.
Yu B. Stability. Bernoulli: official journal of the Bernoulli Society for Mathematical Statistics and Probability. 2013; 19(4):1484–1500. doi:10.3150/13-bejsp14.
Meinshausen N, Bühlmann P. Stability selection. J R Stat Soc Ser B, Stat Methodology. 2010; 72(4):417–73. doi:10.1111/j.1467-9868.2010.00740.x.
Hoeting JA, Madigan D, Raftery AE, Volinsky CT. Bayesian model averaging: A tutorial. Stat Sci. 1999; 14(4):382–417. doi:10.1214/ss/1009212814.
George EI, McCulloch RE. Approaches for bayesian variable selection. Statistica Sinica. 1997; 7:339–73.
Breiman L. Random forests. Machine Learning. 2001; 45(1):5–32. doi:10.1023/A:1010933404324.
Mosteller F, Tukey JW. Data Analysis and Regression - A Second Course in Statistics, Student. edn. Reading, MA: Addison-Wesley Publishing Company; 1977.
Bühlmann P, van de Geer S. Statistics for High-Dimensional Data: Methods, Theory and Applications, 1st edn. Springer, Heidelberg, Germany: Springer Series in Statistics; 2011. doi:10.1007/978-3-642-20192-9.
Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc Ser B, Stat Methodology. 2005; 67(2):301–20. doi:10.1111/j.1467-9868.2005.00503.x.
R Development Core Team. R: A Language and Environment for Statistical Computing. 2014. http://www.R-project.org/. Accessed on 07 February, 2015.
Shankar J. regeval: A systematic evaluation of high-dimensional, ensemble based regression for exploring large model spaces in microbiome analyses. GitHub. 2014. http://github.com/openpencil/regeval. Accessed on 07 February, 2015.
Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York, NY, USA: Springer Series in Statistics; 2013. doi:10.1007/978-0-387-84858-7.
Friedman J, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Software. 2010; 33(1):1–22.
Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Ser B, Stat Methodology. 1996; 58(1):267–88. doi:10.2307/2346178.
Efron B, Hastie T, Johnstone I, Tibshirani R. Least angle regression. Ann Stat. 2004; 32(2):407–99. doi:10.1214/009053604000000067.
Grandvalet Y, Chiquet J, Ambroise C. Sparsity by Worst-Case Quadratic Penalties. arXiv preprint. 2012. http://arxiv.org/abs/1210.2077 Accessed on 07 February, 2015. arXiv:1210.2077 Code: http://r-forge.r-project.org/projects/quadrupen/ Accessed on 07 February, 2015.
Mason DM, Newton MA. A rank statistics approach to the consistency of a general bootstrap. Ann Stat. 1992; 20(3):1611–24. doi:10.1214/aos/1176348787.
Praestgaard J, Wellner JA. Exchangeably weighted bootstraps of the general empirical process. Ann Probability. 1993; 21(4):2053–86. doi:10.1214/aop/1176989011.
Freedman D. A remark on the difference between sampling with and without replacement. J Am Stat Assoc. 1977; 72(359):681–1. doi:10.1080/01621459.1977.10480637.
Bühlmann P, Yu B. Analyzing bagging. Ann Stat. 2002; 30(4):927–61. doi:10.1214/aos/1031689014.
Friedman JH, Hall P. On bagging and nonlinear estimation. J Stat Planning Inference. 2007; 137(3):669–83. doi:10.1016/j.jspi.2006.06.002.
Dümbgen L, Samworth RJ, Schuhmacher D. Stochastic search for semiparametric linear regression models. In: Banerjee M, Bunea F, Huang J, Koltchinskii V, Maathuis MH, (eds.) From Probability to Statistics and Back: High-Dimensional Models and Processes – A Festschrift in Honor of Jon A. Wellner. Beachwood, Ohio, USA: Institute of Mathematical Statistics: 2013. p. 78–90. doi:10.1214/12-IMSCOLL907.
Bach FR. Bolasso: model consistent lasso estimation through the bootstrap. In: Proceedings of the 25th International Conference on Machine Learning. ICML '08. New York, New York, USA: ACM Press: 2008. p. 33–40. doi:10.1145/1390156.1390161.
Scott SL. BoomSpikeSlab: MCMC for spike and slab regression. R package version 0.5.2 CRAN. 2014. http://CRAN.R-project.org/package=BoomSpikeSlab Accessed on 07 February, 2015.
Scott SL, Varian HR. Predicting the present with bayesian structural time series. Int J Math Modell Numer Optimisation. 2014; 5(1/2):4. doi:10.1504/ijmmno.2014.059942.
Brodersen KH, Gallusser F, Koehler J, Remy N, Scott SL. Inferring causal impact using Bayesian structural time-series models. Ann Appl Stat. 2014. In press. http://www.e-publications.org/ims/submission/AOAS/user/submissionFile/17112?confirm=ef7e4bee. Accessed on 07 February, 2015.
Liang F, Paulo R, Molina G, Clyde MA, Berger JO. Mixtures of g priors for bayesian variable selection. J Am Stat Assoc. 2008; 103(481):410–23. doi:10.1198/016214507000001337.
Cortes C, Mohri M. AUC optimization vs. error rate minimization. In: Thrun S, Saul LK, Schölkopf B, Thrun S (eds.) Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press: 2004. p. 313–20.
Das I, Nightingale P, Patel M, Jumaa P. Epidemiology, clinical characteristics, and outcome of candidemia: experience in a tertiary referral center in the UK. Int J Infectious Dis. 2011; 15(11):e759–63. doi:10.1016/j.ijid.2011.06.006.
Smeekens SP, van de Veerdonk FL, Kullberg BJ, Netea MG. Genetic susceptibility to candida infections. EMBO Mol Med. 2013; 5(6):805–13. doi:10.1002/emmm.201201678.
Giri S, Kindo AJ. A review of candida species causing blood stream infection. Indian J Med Microbiol. 2012; 30(3):270–8. doi:10.4103/0255-0857.99484.
Eggimann P, Bille J, Marchetti O. Diagnosis of invasive candidiasis in the ICU. Ann Intensive Care. 2011; 1(1):37. doi:10.1186/2110-5820-1-37.
Woo PCY, Lau SKP, Teng JLL, Tse H, Yuen K. Then and now: use of 16S rDNA gene sequencing for bacterial identification and discovery of novel bacteria in clinical microbiology laboratories. Clin Microbiol Infection. 2008; 14(10):908–34. doi:10.1111/j.1469-0691.2008.02070.x.
Schoch CL, Seifert KA, Huhndorf S, Robert V, Spouge JL, Levesque CA, et al.Fungal Barcoding Consortium, Fungal Barcoding Consortium Author List: Nuclear ribosomal internal transcribed spacer (ITS) region as a universal DNA barcode marker for fungi. Proc Nat Acad Sci USA. 2012; 109(16):6241–6. doi:10.1073/pnas.1117018109.
Szpakowski S. YAP: A Computationally Efficient Workflow for Taxonomic Analyses of Bacterial 16S and Fungal ITS Sequences. GitHub. 2013. http://github.com/shpakoo/YAP. Accessed on 07 February, 2015.
Scott JG, Berger JO. Bayes and empirical-bayes multiplicity adjustment in the variable-selection problem. Ann Stat. 2010; 38(5):2587–619. doi:10.1214/10-AOS792.
May A, Abeln S, Crielaard W, Heringa J, Brandt BW. Unraveling the outcome of 16S rDNA-based taxonomy analysis through mock data and simulations. Bioinformatics. 2014; 30(11):1530–8. doi:10.1093/bioinformatics/btu085.
Shah RD. Modelling interactions in high-dimensional data with Backtracking. arXiv preprint. 2013. http://arxiv.org/abs/1208.1174. Accessed on 07 February, 2015.
Shah RD, Meinshausen N. Random intersection trees. J Machine Learning Res. 2014; 15(1):629–54.
Witten DM, Shojaie A, Zhang F. The cluster elastic net for High-Dimensional regression with unknown variable grouping. Technometrics. 2013; 56(1):112–22. doi:10.1080/00401706.2013.810174.
Article PubMed Central Google Scholar
Prosser JI. Replicate or lie: The need for replication. Environ Microbiol. 2010; 12(7):1806–10. doi:10.1111/j.1462-2920.2010.02201.x.
We gratefully acknowledge our two anonymous referees whose insightful comments substantially improved our manuscript. We thank the authors of glmnet, BoomSpikeSlab, quadrupen, scales, ggplot2, plyr, reshape2, data.table, Hmisc, doMC, corrplot, RColorBrewer, RStudio and the R Core Development Team for their invaluable contributions to statistical computing. We thank Karla Stucker and Andrey Tovchigrechko for their constructive discussions and Reed Shabman for contributing "Reed's mouse" to our manuscript. This research was supported by NIH grants R01AI054928 and R01DE017088 and the Bill and Melinda Gates foundation grant, OPP1017579.
J. Craig Venter Institute, 9704, Medical Center Drive, Rockville, Maryland, 20850, US
Jyoti Shankar, Sebastian Szpakowski, Stephanie Mounaud, Liliana Losada & William C Nierman
Los Angeles Biomedical Research Institute at Harbor, UCLA Medical Center, 1124 West Carson Street, Torrance, California, 90509, US
Norma V Solis, Hong Liu & Scott G Filler
David Geffen School of Medicine, University of California at Los Angeles, California, 90095, US
Scott G Filler
Jyoti Shankar
Sebastian Szpakowski
Norma V Solis
Stephanie Mounaud
Liliana Losada
William C Nierman
Correspondence to Jyoti Shankar.
JS designed and implemented the simulations, performed the evaluation experiments and wrote the manuscript. SS designed and implemented the YAP bioinformatics workflow for taxonomic assignment and provided crucial insights into biases and structure of microbiome data. NVS and HL conducted the mouse experiments. SM conducted the microbiome sequencing experiments. SGF, WCN, NVS, LL and SM designed the mouse microbiome study. All the authors read and approved the manuscript.
Median values of performance metrics (median_scores.pdf). Figures showing median values of performance metrics presented as boxplots in the main paper.
Performance without log-transformation (no_log_transformation.pdf). Figures showing performance of all the approaches on data without log-transformations.
Performance with weighted inclusion probabilities (weighted_ip.pdf). Figures showing performance of SS, SR, SSW and SRW approaches.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Shankar, J., Szpakowski, S., Solis, N.V. et al. A systematic evaluation of high-dimensional, ensemble-based regression for exploring large model spaces in microbiome analyses. BMC Bioinformatics 16, 31 (2015). https://doi.org/10.1186/s12859-015-0467-6
Received: 08 July 2014
Microbiome analysis
Sparse ensemble-based regression
Stability selection


text:
Brooklyn's Nation of Language to celebrate Album Release nov. 4 at Market Hotel, discuss new…
Steve Malinski	 Nov 4, 2021
Early birds at this year's tenth-annual Governors Ball Music Festival were treated to a high-energy performance to kick off the festival's second day. That band, Brooklyn trio Nation of Language, deliver a fresh take on the synth-popIt wouldn't quite be a complete outdoor music season in New York City without an installment of The Governors Ball Music Festival. After some difficulties with weather cancellations in 2019, GovBall had been looking forward to celebratingFounders Entertainment has announced the lineup for the 2021 edition of Governors Ball Music Festival. The NYC-based festival celebrates its 10th Anniversary and is set to take place Friday, September 24 through Sunday, September 26 at theGovernors Ball Music Festival announced it's 2021 dates tentatively scheduling its 2021 edition for September 24-26. The festival generally takes place in June but is optimistically hoping by September that COVID-19 will be under wrapsOur annual recap of the Best of 2019 has arrived at music festivals. Just like when fans arrive at these ever-popular weekend extravaganzas, we're bright-eyed and bushy-tailed looking forward to what surprises lay around the corner. OurGovernors Ball may be moving to the Bronx in the future, relocating from Randall's Island to Van Cortlandt Park in the Bronx. While the festival has been denied a permit to move locations in 2020, the company behind the festival made a The Governors Ball Music Festival, held annually on Randall's Island in New York, is instituting an 18 and over age policy for festival attendees beginning in 2020. The 10th edition of the festival will be held June 5-7. The Yeah YeahThis past weekend, Randall's Island played host to the annual Governors Ball Music Festival. The festival, in its ninth year, featured many of the hottest musical acts of today with performances by Tyler the Creator, Florence and TheThe Governors Ball Music Festival will hold its 9th event over May 31-June 2, taking place at Randall's Island Park in New York City. The impressive line up includes Tyler, the Creator, Florence + the Machine, Kacey Musgraves, Charlie XCX,

redpajama_set_name: RedPajamaCommonCrawl


text:
package cz.jirutka.spring.exhandler.handlers;

import cz.jirutka.spring.exhandler.messages.ErrorMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.ResponseEntity;
import org.springframework.web.servlet.DispatcherServlet;
import org.springframework.web.servlet.mvc.multiaction.NoSuchRequestHandlingMethodException;

import javax.servlet.http.HttpServletRequest;

import static org.springframework.http.HttpStatus.NOT_FOUND;

public class NoSuchRequestHandlingMethodExceptionHandler
        extends ErrorMessageRestExceptionHandler<NoSuchRequestHandlingMethodException> {

    private static final Logger LOG = LoggerFactory.getLogger(DispatcherServlet.PAGE_NOT_FOUND_LOG_CATEGORY);

    public NoSuchRequestHandlingMethodExceptionHandler() {
        super(NOT_FOUND);
    }

    @Override
    public ResponseEntity<ErrorMessage> handleException(NoSuchRequestHandlingMethodException ex, HttpServletRequest req) {

        LOG.warn(ex.getMessage());
        return super.handleException(ex, req);
    }
}

redpajama_set_name: RedPajamaGithub


text:
Aevee Bee and Mia Schwartz Talk "We Know the Devil" and "Heaven Will Be Mine"
by Brenda Noiseux | Mar 27, 2017 | Features | 0 comments
Amidst the din of the Sunday after show floor, I headed to the Visual Novel Reading Room in the Indie Megabooth at PAX East to meet up with Aevee Bee and Mia Schwartz, creators of the indie visual novel game, We Know the Devil. In typical go-with-the-flow PAX fashion, we hold our interview standing around a nearby pillar, while I quietly hoped my voice recorder would hold up to the noise. Over the next half hour, we chatted about the new We Know the Devil demo for PAX East, their upcoming project, Heaven Will Be Mine, and pushing the boundaries of visual novel games.
We Know the Devil has been out for a while and has received wonderful critique and acclaim. Looking back, what do you think of this wild ride?
Schwartz: Mostly I forget how much we accomplished with it. For me, finishing it had me grow so much as an artist and art director. Looking back now these drawings are kind of wonky, but that's good. It means you're getting better.
Bee: It feels really nice. We wrote a demo for the [PAX East] showroom floor, the idea was to have a very short 5-10 minute demo that you can read in one sitting. That allowed me to review the characters and write them again. It was really fun to explore sides of them I hadn't really had the chance to in the game. And there were things, from the reactions of fans, of people seeing these characters in these particular ways. With the demo, I leaned some of the characters in other directions that I thought were important to their character but were under-emphasized in the original game.
Schwartz: Yes, there were some subtleties in the main game that the demo underlines. I was pretty emotional reading the script because it was so nice to hear their voices again. I didn't do any new art for the demo; they just used my old sprites from the game. And it had been so long since we had used them. There are all these facial expression choices that were used for the opposite reason they were drawn, but they work really well. I had nothing to do with the decisions. It was kind of amazing to see and it made them feel even more alive than in the original game. I feel really good about coming back here. Feeling really good about having something new with We Know the Devil as we just announced our next game.
Is the demo at PAX East completely new?
Bee: Yes, it's totally new content. We'll put a more polished build up online sometime soon. For the show, we needed an original self-contained thing. It's very short that takes place a couple hours before the original game starts. It's more reason to get people in as well as for old friends.
Schwartz: There's a lot of excitement that's been generated just for people to see more of them. More than I expected.
Are you thinking of doing anything else for We Know the Devil?
Bee: We thought about doing some illustrated prose or something like that. But I think we're done with games with those specific characters since it's a very self-contained small story so I don't see there being too much in those particular characters in that particular story. Our follow-up project discovers that world from very different angles, like a bigger picture that connects all the fabric.
Schwartz: There are literal and figurative connections that happen. Overarching themes, figures or concepts will continue on and intersect with other things we're doing.
So what is the next project?
Bee: The next project will be called Heaven Will Be Mine. We have some postcards but we don't have too much for it outside a short key visual, short description of it…
Schwartz: We have a year's worth of design work…
Bee: Well, that's all we have to announce right now.
Schwartz: We're taking it really slow for a couple reasons but we're hoping to have it out this year. [looks to Bee] Do you want to talk about the premise?
Bee: It's kind of like a thematic sequel. If We Know the Devil is about teens first discovering feelings, then Heaven Will Be Mine is about 20-somethings living with that. The premise is, there are a bunch of giant robot pilots who are fighting each other and falling in love and flirting with their enemies and the people with they're supposed to be defeating in this space conflict.
I love it. I love it.
Bee: We went through like several different concepts of this game and we're arriving at something more robust. Since we wanted to hit the ground running, we thought it was time to make an announcement to let people know about it.
When I was playing We Know the Devil it felt very middle school, very teen, very young, did you get feedback from players who wanted something a little bit older? What made you go in the order of the younger protagonists first?
Bee: That's a very good question. I've talked a little bit about the origin for the idea of We Know the Devil. I wanted it to to have a sort of Sailor Moon or Buffy vibe to it. There were these young queer feelings I wanted to talk about from this angle, but it's also a story that gets told over and over a lot. There are so many coming out stories, but there are very few stories about people living their lives as queer adults. And I find that paints a very fraught and incomplete picture.
Schwartz: I also, I can't really speak for Aevee, but We Know the Devil is a very small game we decided to make for a game jam initially. It was supposed to be this tiny thing. And then I was like "Fuck it, I guess we're making a game." At least from my perspective, Aevee kind of pitched this idea to me of let's do a cool story about teens getting possessed by the devil, for us. In my tiny bubble, I don't make work for kids or teenagers, and I don't have them in my sphere but somehow it did not occur to me, that it would get so big that actually reach teens. [awkward laughter]
There's not a lot, but there are maybe some things I wouldn't have done if I had thought at the time that a teen would find it. I stand by the game, but there's some stuff that is different for a 20 year old to read what these kids are doing than for a 14 year old. And that's something that I've had to internalize in terms of what I draw characters doing or saying. I've seen some kids, it's none of my business what they do, but I have seen them pick up maybe bad habits from my drawing.
That's really interesting to know because sometimes we forget other audiences might pick this (game, comic, novel, etc) up and their could be unintended consequences. It's a challenge everywhere.
Bee: It's a real difficult sort of thing, wanting to balance creating works that are responsible and honest that speak to the experience, and being responsible to the audience that may experience it. I am actually really interested in making work for young adults. When I was a creative writer, that was a big part of my focus and that I loved. There was a lot of stuff as a kid that I was not patronized by, which was really important and a bit of what I was going for with We Know the Devil. With Heaven Will Be Mine, it's nice to make something about my own life that I've been living for a while. I think we'll probably be doing the same sort of thing, like thinking about our responsibilities with it, especially because they're messier, more involved.
Schwartz: There's more um, gray areas, and more um–
Complexity?
Schwartz: Exactly, there's kind of imagery, analogies and metaphors that are not translated super easily and are existing as multiple things at once. It's been an interesting thing to balance working on this, for us to figure out like where does our responsibility lie and how does that intersect with what we want to say and the kind of people we would want to reach and the kind of message that some people need that would be communicated by not pulling punches and not not getting messy.
That sounds like a really large challenge.
Schwartz: Yup.
Bee: It's very difficult to do that, so all we can do is to try our best, but I feel like with We Know the Devil our honesty and the nuance of that really came forth in a really big way. Sometimes you find yourself with fans who are just learning stuff of themselves and they're getting challenged and they're finding themselves getting pulled all over the place by different messages. We can only hope that this will be a positive chunk in a complete world picture that maybe not be complete yet. And maybe not complete for a while. Or maybe never complete b/c we're like older but not like that much more put together.
Will the latest project have a similar art style to We Know the Devil? Or are we changing things up?
Schwartz: I'm not very good at masking the kind of things I draw inherently. I will say the art direction is maybe the complete opposite. I'm feeling really good about it because hopefully there's going to be a lot of experimentation and kind of mixed medium. I really feel good how it will look next to We Know the Devil, I think the contrast will work really well. But I'm really excited to do the complete opposite. A lot of the choices I've made for the look of We Know the Devil were intentional, not just for communicating the world and tone we wanted, and to call on the influences we had, but also because we didn't have a lot of time. I was not super confident in my ability to make something slightly consistent because I'm very messy. And that's not what a visual novel looks like. And I feel good that there's not been a visual novel that looks like We Know the Devil.
Bee: It's been one of the things I initially came to this with is that I didn't want it to look like other visual novels. Mia comes from a comics background and has a visual storytelling. She does so much visualized storytelling and characterization through her images, which is I think unique, especially because a lot of visual novels have a very pretty detailed still image, but it's stilted and static, whereas Mia's illustrations have a lot more life and an energy to them. The thing that we're doing with Heaven Will Be Mine is that it's played through a cockpit screen, so you're actually looking through a computer interface. The way that the action-y scenes, the way the romantic encounters happen you'll see a ton of pop-up windows that will depict what's happening between the robot's fighting, what the characters themselves are experiencing and then this mixed media abstract. Which I think is just an amazing concept for the visual style. I'm stoked about that because very few visual novels deviate from that particular format.
It's great that you're pushing barriers, not only with the storytelling aspect but the actual art that visual novels have become. That's one of the things I personally like about We Know the Devil. It does stand out and it's not like we had a graphic novel that we turned into point-and-click. There's more game and not just novel.
Bee: I think about the whole narrative design how do we just like use the choices and techniques available to us within a game to better tell a story. We Know the Devil has a very simple scary mechanic but it's immediately interesting and understandable to the player. You can just tell someone the premise and they're like "I understand what it's about" and it communicates the central theme of the game so well. We're working on, and feel we're pretty close, to to finding something we're satisfied with for Heaven Will Be Mine. Something that will do similar in a different kind of storytelling.
Schwartz: A huge part of the design in our games comes from the fact that we personally don't like the lack of agency that some choices in visual novels have. Where it's like one choice after so much text and you don't actually know the kind of impact it will have on the story. The whole point of a video game versus something is the act of choosing something makes the player feel something in itself. That's something that Aevee does an incredible amount of design to make sure it doesn't feel like that. To make sure there's intention and agency.
And that something down the road will change because of these decisions that the player has made.
Schwartz: I also appreciate Ladykiller in a Bind for just how beautiful and functional and unique the actual mechanic is. For your choices, it says what's going to happen, but not in a way that doesn't remove you from; it says what kind of moving parts, like it unlocks this room or like you'll get this vote, but not in a way that detracts from the experience from making that choice.
Bee: Ladykiller in a Bind uses this cost system, like you might get in trouble for some choices or you might gain for other choices and that can get used to let players make the decisions. I can make this decision but I might get in trouble making this decision. It's kind of like, we had explored similar things in this game as well, to help guide the player to where they make certain kinds of choices. Or even just having to think about their choices even if the effect is really no different with or without the tag, or warning, from the game. But that will make them think about their choices differently which is also part of the storytelling.
Kind of like the real world, I could decide not to go to work today but …
Bee: Right. You know exactly what's going to happen. Whereas a lot of visual novels and dialogue-based games are like, you're going to say this. You have no idea.You have one line of dialogue. How do you even what effect that's going to have, you really only have the vaguest idea. Not very interesting for the player.
Schwartz: Another thing, personally one of the things for me, I really like the sub-visual novel format that's present in more recent persona games. Just in the fact that you pick. There's this element of "I'm interested in what's going on here" and I have the freedom to kind of check all these little worlds out and it doesn't affect the greater thing. I can just have this small moment. Or I want to see what happens when I mash these two dolls together and that's sort of like, a similar feeling to the mechanic of We Know the Devil. Like what are these characters going to do in this situation, I really want to see these ones versus these other ones. And that's something I feel proud of making.
We Know the Devil is an indie game about teens fighting the devil at summer camp. In this group horror, one person is always left out. The game was released in 2016 and reviewed here by Games Editor Al Rosenberg. It is currently available on Steam. Heaven Will Be Mine is expected to be out later this year. Follow updates at the game's website.
Brenda Noiseux
Community builder, artist, convention organizer, gamer, geek writer Women Write About Comics and Sidequest. Product Maven at Almost a Game. Owner, Bittenby Studios.


Based on a photograph by Ronnie Cutrone.
Bears the papermaker's chop mark.
Based on the print F&S II.164, the portfolio illustrates the five separate colors of the print progression (II.165-II.169) plus one (II.171) with only the drawing line and one (II.170) with the black separation in registration with the drawing line.
This portfolio is Warhol's only published progressive print series which specifically illustrates the breakdown of the printmaking process.


An Invitation to Rediscover the Lord's Supper This class runs May 14 to June 11 and is taught by Dr. John Burgess.
Imagine the disciples sharing meals with Jesus, now risen from the dead. Now Jesus' words and actions at that final Passover supper before his arrest have a whole new meaning. We, too, share the table with the resurrected and ascended Christ every time we partake of the Lord's Supper. This Sunday morning adult class will focus on how the sacraments call us into deeper communion with God, with each other in the church, and with people in the world around us. We will look especially at the Lord's Supper and the structure of the church's communion prayers, and we will discuss how we can participate in communion with greater joy and faithfulness.
Sunday morning discipleship classes are opportunities to grow through Bible study and learning about theology, church history, spiritual formation, faith and life issues, and ministry skills and tools. Nursery and Rock Club for K-5th grade happen at the same time so parents can enjoy their own time for fellowship and study.


A series of love poems by Páll Ólafsson named Eg skal kveða um eina þig alla mína daga (roughly translated; I will write of only you for the rest of my days). The title is impressed on a red, loose leather which covers the creamy pages within and everything is held together with a thick, texture-rich seam.
To lend the poems a warm personal feeling, the beautiful and humanistic serif-typeface Adobe Jenson was applied throughout the piece.


We have found this the most successful way of finding new talent, having signed authors who have seen our presentations at events such as KidLitVIC, CYA Conference, Emerging Writers' Festival, and the ACT Writers Centre Hard Copy Fiction and Non-fiction programs.
I am not accepting Young Adult (YA) or Middle Grade (MG) submissions in 2019. This may change, but for now I have a full list and am focusing on those authors I am currently representing. I will be open to receiving more YA and MG works – possibly in late 2019 – but check back here for updates!


text:
Home › Search Resources (English): Aboriginal peoples
Search Resources (English): Aboriginal peoples
A fact sheet on Aboriginal health research governance
http://www.cwhn.ca/sites/default/files/PDF/CEWH/AWHHRG/factSheet06.pdf
Examines Aboriginal research ethics review processes, research agreements and models for ensuring that research is directed by and accountable to Aboriginal communities.
(See Details)
Report on the regional consultation held in Kahnawake, Quebec
http://www.cwhn.ca/sites/default/files/PDF/CEWH/AWHHRG/consultationKahnawake.pdf
Discusses AWHHRG's bibliography, with such themes as Aboriginal research methodology, gaps in existing research, qualitative and quantitative research, and strategies and methods.
Cultural competence for registered nurses
Details recommendations to ensure appropriate cultural competence nursing care for Aboriginal peoples.
Mother launches campaign to recruit Aboriginal bone marrow donors
Describes the work of one woman, Gloria Greyeyes, who is raising awareness for the need of Aboriginal bone marrow donors.
Living in balance: gender, structural inequalities, and health promoting behaviors in Manitoba First Nation communities
http://www.cewh-cesf.ca/PDF/pwhce/livingInBalance.pdf
Describes health promoting behaviours such as engaging in more physical activity, positive dietary changes, quitting smoking, and stopping drinking for a time, in First Nations peoples.
Reading Romanow : the implications of the final report of The Commission on the Future of Health Care in Canada for women
http://www.cewh-cesf.ca/PDF/health_reform/reading-romanow.pdf
Offers a chapter-by-chapter gendered analysis of the Final Report of the Royal Commission on the Future of Health Care, Building on Values: The Future of Health Care in Canada (Romanow 2002).
Journal of Aboriginal Health
Shares success stories in Aboriginal health, discusses issues and opportunities, and provides the latest information and research of interest to First Nation, Inuit and Métis Peoples.
Published: 2004-
Canada health action: building on the legacy: volume II: synthesis reports and issues papers
Presents the detailed analysis and rational, as well as recommendations, that led to the forum's priorities, outlined in Volume I.
Position of Quebec Native Women on the Canadian ratification of the American Convention on Human Rights and the Inter-American C
http://www.faq-qnw.org/convention-americaine/position_paper_am_conv.pdf
Recommends that Canada ratifies these two conventions in order to lend its weight to the protection and promotion of the human rights of Aboriginal peoples in the Americas.
Reducing health disparities and promoting equity for vulnerable populations. Aboriginal Canada: synthesis paper
Reivews the health disparities among Aboriginal peoples (First Nations, Inuit and Métis) in Canada, all direct and indirect results of social, economic, cultural and political inequities.
Subject/Terms
: all » Aboriginal peoples
Centres of Excellence for Womens Health (CEWH) (6)
Prairie Women's Health Centre of Excellence (PWHCE) (3)
British Columbia Centre of Excellence for Women's Health (BCCEWH) (2)
Canadian Women's Health Network (CWHN) (2)
National Network on Environment and Women's Health (NNEWH) (1)
Women and Health Care Reform (1)
Women's Health New Genetics (WHNG) (1)
: Aboriginal peoples


Funk Styles or funk dance refer to dance styles that are primarily danced to funk music. More specifically they can be used for a group of street dance styles that originated in California in the 1970s, mainly popping and locking. Though these specific dance styles have today been incorporated into the hip hop culture to some extent, and are often seen danced to hip hop music and electronica as well, they were originally and are still commonly danced to funk music. One of the reasons that the term funk styles first appeared was to give these dances their own identity and avoid them being primarily associated with hip hop and breakdancing.
In the 80's when streetdancing blew up, the media often incorrectly used the term "breakdancing" as an umbrella term for most the streetdancing styles that they saw. What many people didn't know was this within these styles, other sub-cultures existed, each with their own identities.
Breakdancing, or b-boying as it is more appropriately known as, is known to have its roots in the east coast and was heavily influenced by break beats and hip hop. The term "funk styles" was coined to give what we do it's own identity and separate it from hip-hop. Popping, locking and boogaloo were styles that were created in the WEST COAST during the FUNK ERA, and while these styles were adopted into the hip-hop movement, its roots should still be recognized as pure funk. Hence the term, FUNK STYLES.
In a town called Fresno, California, there lived a shy boy named Sam. Inspired to create his own style of dance after seeing the original Lockers perform on TV, in 1975 Sam started putting together movements which later became known as boogaloo or boog style.
The name came from the old James Brown song "Do the Boogaloo". One day when Sam was dancing around the house, his uncle said "Boy, do that boogaloo!" A puzzled Sam asked his uncle, "What's boogaloo?". "That means you're gettin down" his uncle replied. From that day on he was known as Boogaloo Sam.
Not many people know what boogaloo style is or how to do it. Boogaloo is a fluid style that uses every part of the body. It involves using angles and incorporating fluid movements to make everything flow together, often using rolls of the hips, knees, head. Making your legs do wierd things, and covering a lot of space on stage using "walkouts" or other transitions to get from one spot to the next spot. Although it is described as fluid, please note that boogaloo is different from the style known as waving.
A lot of people ask what Electric Boogaloo style is. Electric Boogaloo style is combining popping and boogaloo style together. The two styles compliment each other well and is known worldwide as the signature style of the EB's.
While Sam was creating popping and boogaloo, others were creating and practicing unique styles of their own. Back in the day many different areas in the west coast were known for their own distinct styles, each with their own rich history behind them. Some of these areas included Oakland, Sacramento and San Fransisco.
Although the EB's primarily pop and boogaloo, we still like to mix it up and encourage all dancers out there to learn and mix other styles as well.
The Electric Boogaloos would like to give respect to the other OGs and innovators of all styles who contributed to make the street dance scene what it is today. Keep the funk alive!
After seeing legendary group the Lockers perform on television, Sam was inspired to create his own dance style. Around the years of 1975-1976 Sam created a set of movements that evolved into the styles known today as popping and boogaloo (boog style). In 1977 Sam founded the Electronic Boogaloo Lockers, who later became known as the Electric Boogaloos.
Currently Sam is still getting down and is the active leader and member of the EB's. A true innovator of funk styles, Sam has helped push the boundaries to where they are today.
Tutting is the name given to a contemporary abstract interpretive dance style that exploits the body's ability to create geometric positions and movements, predominantly with the use of right angles. It is performed with minute attention to the musical rhythm, typically by altering positions in a stop-and-go fashion.
Both boxing and the hinge illusion are special applications of the mime concept fixed point. Another style, called fixed line, requires that dancers apply this concept rigorously to more convincingly convey the illusion of solid shapes. Much as a mime conveys a wall by always keeping one hand on the wall, or shows a rope by always keeping one hand on the rope, a tutter shows a shape by always maintaining at least one side of the shape. To do this, a tutter will use his body parts to assemble a shape segment by segment and dissasemble it in the same piecemeal fashion.


Impinj recently launched their new xArray, ceiling-mounted RFID reader system, intended to deliver "always on item intelligence." Here we consider the industry implications and why it is a meaningful step forward for the RFID industry.
Impinj laid down an important milestone last week on the road to fully realizing the benefits of RFID. After a lengthy "Early-Access Program" involving over a dozen partners, Impinj launched their xArray, ceiling-mounted reader system, which is capable of steering a full-power, UHF RFID reader beam around a circle with a 7-meter radius.
The xArray incorporates a Speedway Revolution reader, a multi-element, steerable antenna system and special software algorithms in a deployment-friendly, self-contained, overhead-mounted appliance. Applications include retail inventory management, tracking assets such as hospital equipment and tracking manufacturing work in progress (WIP).
In retail, xArray stands on the shoulders of the increasingly common RFID handheld inventory management application (see The Evolution of Retail Store RFID Reader Strategies). Simplicity, low capital cost and large return on investment (ROI) are currently driving rapid adoption of the handheld solution. Compared to barcode-based inventory counting, handheld RFID offers substantially more accurate inventory accuracy leading to big jumps in revenue and gross margin from reduced out-of-stocks and markdowns. Labor savings can be substantial, although many retailers cite better customer service rather than cost reductions. Based on primary research with retailers and extensive financial analysis, we estimate typical payback periods for handheld solutions of 9-18 months, and an ROI that any CFO will love: highly-positive net-present-values (NPV) and three-digit internal-rates-of-return (IRR).
But handheld solutions fall well short of the full potential of RFID. For one thing, handheld RFID depends on employee performance and compliance. One of the early learnings from field trials was that inventory accuracy was highly variable and dependent on how effectively the handheld was used, and as a result, staff training and compliance were critical. Garbage in, garbage out.
While fixed infrastructure has the potential to deliver significantly more value than handhelds, it cannot compare with handhelds on a capital expense basis (see Comparing the Economics of Fixed vs. Handheld RFID Reader Options). The hardware for the 8,000-square-foot American Apparel pilot store in Melrose, California, with its 15 readers, 150 antennas and myriad cables, carries an MSRP list price of over $45K, excluding a substantial installation cost.
But xArray should significantly reduce the capital hit for fixed infrastructure. With its 1,500-square-foot coverage area and $3,300 MSRP, xArray could cut the capital outlay for overhead monitoring by more than half, and as a self-contained, power-over-Ethernet (POE) unit, deployment will be much less expensive. This comparison doesn't address differences in how often tags are read, but for many apps, such as inventory management, it doesn't matter as long as you read a tag a few times per day.
xArray is just an appliance, and making good on this vision will require a complete solution. According to Impinj, their Early-Access Program partners have been working on software and solutions to take advantage of the xArray for about a year. Moreover, Impinj messaging seems to foreshadow a new, software-rich strategy to follow-through on the "Always On" vision. From what we've seen, the ongoing development of the software brains behind item-level intelligence will be one of the most significant strategic shifts in Impinj's history. Look forward to hearing more about it from us in 2015.