title: Estimate degrees of freedom in linear mixed model contrasts on large datasets using R
body: <p>I have a large dataset that I have been analysing using a linear mixed model (~600000 observations). Usually, I estimate the least-square means and do pairwise contrasts using the <strong>lsmeans</strong> from <strong>lsmeans</strong> package. I also use the estimates of the degrees of freedom to test for the significance of each contrast, using  a <em>t</em> statistic, as many times I have actually a quite low number. However, I cannot apply <strong>lsmeans</strong> on my dataset. Below I show example, in which I simulate datasets of different sizes.</p>
  
  <pre><code>library(lme4)
  library(lsmeans)
  
  SmallData&lt;-data.frame(R=rnorm(10000,23,4),A=sample(letters[1:2],10000,replace=TRUE),B=sample(letters[11:23],10000,replace=TRUE))
  MediumData&lt;-data.frame(R=rnorm(100000,23,4),A=sample(letters[1:2],100000,replace=TRUE),B=sample(letters[11:23],10000,replace=TRUE))
  LargeData&lt;-data.frame(R=rnorm(600000,23,4),A=sample(letters[1:2],600000,replace=TRUE),B=sample(letters[11:23],10000,replace=TRUE))
  
  modelSmall&lt;-lmer(R ~ B + (1|A),data=SmallData)
  modelMedium&lt;-lmer(R ~ B + (1|A),data=MediumData)
  modelLarge&lt;-lmer(R ~ B + (1|A),data=LargeData)
  
  Small.lsm&lt;-lsmeans(modelSmall,~B)
  
  Small.lsm
  
  $lsmeans
   B   lsmean         SE    df lower.CL upper.CL
   k 22.99691 0.05649338 74.85 22.88436 23.10945
   l 22.99451 0.05656906 75.68 22.88184 23.10719
  
  Results are averaged over the levels of: C
  Confidence level used: 0.95
  
  $contrasts
   contrast    estimate        SE      df t.ratio p.value
   k - l    0.002397375 0.0799509 9992.92    0.03  0.9761
  
  Results are averaged over the levels of: C
  
  Small.lsm&lt;-lsmeans(modelSmall,pairwise~B|C)
  
  Small.lsm
  
  $lsmeans
  C = m:
   B   lsmean         SE     df lower.CL upper.CL
   k 22.98629 0.07980879 294.10 22.82923 23.14336
   l 23.01980 0.07946976 286.64 22.86338 23.17621
  
  C = n:
   B   lsmean         SE     df lower.CL upper.CL
   k 23.00752 0.07996624 292.46 22.85014 23.16490
   l 22.96923 0.08053307 304.92 22.81075 23.12770
  
  Confidence level used: 0.95
  
  $contrasts
  C = m:
   contrast    estimate        SE      df t.ratio p.value
   k - l    -0.03350181 0.1126268 9996.00  -0.297  0.7661
  
  C = n:
   contrast    estimate        SE      df t.ratio p.value
   k - l     0.03829656 0.1134942 9994.74   0.337  0.7358
  
  
  Medium.lsm&lt;-lsmeans(modelMedium,pairwise~B|C) # Memory overload (~70 Gb RAM)
  
  Large.lsm&lt;-lsmeans(modelLarge,pairwise~B|C)
  
  Large.lsm&lt;-lsmeans(modelLarge,pairwise~B|C)
   *** caught segfault ***
  address 0x7f9de00ff000, cause 'memory not mapped'
  
  Traceback:
   1: t(ZZ) %*% EE %*% ZZ
   2: t(ZZ) %*% EE %*% ZZ
   3: .get_SigmaG(object, details)
   4: get_SigmaG.lmerMod(object, details)
   5: get_SigmaG(object, details)
   6: pbkrtest::vcovAdj.lmerMod(object, 0)
   7: lsm.basis.merMod(object, trms, xlev, grid, ...)
   8: lsm.basis(object, trms, xlev, grid, ...)
   9: ref.grid(object = &lt;S4 object of class "lmerMod"&gt;, by = "C", contr = "pairwise")
  10: do.call("ref.grid", rgargs)
  11: lsmeans.default(object, specs, ...)
  12: lsmeans.character.default(object, specs = all.vars(specs[-2]),     by = by, contr = contr.spec, ...)
  13: lsmeans.character(object, specs = all.vars(specs[-2]), by = by,     contr = contr.spec, ...)
  14: lsmeans(object, specs = all.vars(specs[-2]), by = by, contr = contr.spec,     ...)
  15: lsmeans.formula(modelLarge, pairwise ~ B | C)
  16: lsmeans(modelLarge, pairwise ~ B | C)
  </code></pre>
  
  <p>I have also tried <strong>lmerTest</strong> package and used <strong>difflsmeans</strong> function and it also crashed. I can get the mean estimates using general linear hypothesis testing - <strong>glht</strong> from <strong>multcomp</strong> package -, but this procedure does not estimate degrees of freedom of the specific contrasts. Do you know of any alternative to these procedures that is applicable to large datasets?</p>
  
  <p>Thank you very, very much.</p>