Select polynomial order for continuous variables in mixed model step-wise backward selection I am working on some data that I would like to analyze through a generalized linear mixed model regression and a stepwise backward selection of variables directly on that model.
I use the GLMERSelect R package which seems developed exactly for this purpose. In this function, you specify the response variable (responseVar), model family (fitFamily, binomial in my case) random effect variables (randomStruct), fixedTerms (continous variables) and fixedFactors (categorical variables). You can find a nice usage examples here.
As you can see from the example, for the continuous variables you specify the polynomial order that you should use, in the example they use 2 fixedTerms = list(logHPD.rs=2,logDistRd.rs=2)
My theoretical question is: how do you choose this order for each continuous variable? Why and when would you use a quadratic (order=2) instead of a linear (order=1) or a higher order polynomial?

A single polynomial fitted to data in regression can be a recipe for distaster unless you have a strong theoretical justification for a particular polynomial form. This page among others explains many reasons why.
A better general approach is to use a smooth continuous model of a continuous predictor, a model doesn't impose a single functional form over its entire range.  This page outlines different ways to accomplish this.
For smooths in generalized additive models or smoothing splines, you can adjust the magnitude of penalization to best fit your data. Regression splines specify a number of "knots" spaced along the continuous values, fitting a cubic polynomial individually between each pair of knots in a way that joins the individual fits smoothly at each knot. Increasing or decreasing the number of knots adds or subtracts from the number of coefficients to estimate. With regression splines you can thus choose the number of knots, perhaps via backward elimination of the type you propose, to adjust the complexity of the fit to the data at hand.
