text_combined: jens oehlschlägel jensoehlschlaegeltrueclustercom dear r community ff version 20 available cran based paging concepts version 10 20 major redesign package handling large datasets implemented numerous enhancements performance improvements make package suitable base package large data processing ff package provides atomic data structures stored disk behave almost ram transparently mapping section pagesize main memory effective virtual memory consumption per ff object addition double data type ff objects support logical raw integer atomic datatypes plus closetoatomic types like factor posixct custom closetoatomic types addition fast vector access ff native support matrices arrays flexible dimorder major columnorder major roworder generalizations arrays raw data still gets stored binary flat files native encoding ff objects extended carry meta information physical virtual attributes ff objects welldefined hybrid copying semantics gives rise certain performance improvements virtualization new ff objects stored reopened across r sessions flat files shared multiple ff r objects using different data endecoding schemes process multiple r processes exploit parallelism wide choice finalizer options allows work permanent files well creatingremoving temporary ff files completely transparent user certain osfilesystem combinations creation process large atomic data sets speedup dramatically using sparse file allocation several access optimization techniques hybrid index preprocessing virtualization implemented achieve good performance even large datasets example virtual matrix transpose without touching single byte disk reduce disk io atomic data gets stored native compact binary flat files ie logicals take exactly 2 bits represent true false na beyond basic access functions ff package also provides compatibility functions facilitate writing code ff ram objects support batch processing ff objects eg asram asff ffapply package supports convenient processing large ff objects making rff make bigger part rs basic functions available ff objects method dispatch andor evaluator handles expressions contain ff objects note professional extension available authors integrates additional highperformance features neatly ff package extension allows efficient handling symmetric matrices supports packed data types boolean 1 bit quad 2 bit unsigned nibble 4 bit unsigned byte 1 byte signed nas ubyte 1 byte unsigned short 2 byte signed nas ushort 2 byte unsigned single 4 byte float nas example quad allows efficient storage genomic data atgc factor unsigned types support circular arithmetic ps interested ff 20 might want visit presentation august 5th jsm highperformance processing large data sets via memory mapping case study r c official package presentation user2008 dortmund scheduled august 13th ff authors daniel adler christian gläser oleg nenadic jens oehlschlägel walter zucchini _______________________________________________ rpackages mailing list rpackagesrprojectorg httpsstatethzchmailmanlistinforpackages ______________________________________________ rhelprprojectorg mailing list httpsstatethzchmailmanlistinforhelp please read posting guide httpwwwrprojectorgpostingguidehtml provide commented minimal selfcontained reproducible code mon 04 aug 2008 101256 0200 r rpkgs major release ff 20 large atomic objects