title: compressing similar columns when converting a data.table from long to wide
body: <p>I am working with a longitudinal data set with repeated observations for subjects in a long format data.table. Most subjects have a few (&lt;10) repeated observations, while a few subjects have many (>100) observations. I can convert this data set from long to wide as below, but it becomes extremely wide (I have many variables at each time point) and is mostly full of NAs, since most subjects don't have data for variables at times 11 through 100. Is there a more elegant way to recast this data to wide format? I'm thinking something along the lines of a ragged array in other languages... </p>
  
  <p>Some solutions exist <a href="https://stackoverflow.com/questions/16596515/aggregating-by-unique-identifier-and-concatenating-related-values-into-a-string">here</a>, but a big concern for me is object size: the wide matrix with lots of NAs takes up a lot of unnecessary space.</p>
  
  <p>A MWE with my current (undesirably sparse matrix) solution is below. Ideally, if some sort of ragged list approach is feasible, the resulting object would have 3 rows and 3 columns, where the "year" and "code" columns are lists or similar. As a bonus, it would be wonderful if I could nest the "code" variable inside the "year" variable as nested ragged arrays. </p>
  
  <pre><code>library(data.table)
  
  dat &lt;- data.table(id=c(rep(1,5), rep(2,10), rep(3,85)),
      year=sample(2013:2016, 100, replace=TRUE),
      code=sample(LETTERS, 100, replace=TRUE))
  
  wideDat &lt;- dcast(dat, id~paste0("code", dat[,seq_len(.N), by=id]$V1), 
      value.var="code")
  </code></pre>