Q: c# Best type/collection/list/dataset to handle super large data (csv/tab files) I am building one WPF (MVVM) app that handles really large csv files. We are talking about 1GB to 10GB.
I open the file and parse it with File.ReadLines into a List of following class:
public class FileLine
{
    public DateTime Time { get; set; } 
    public string Message { get; set; } //Usually around 256 characters
    public string Info1 { get; set; } //Exact 56 characters
    public string Info2 { get; set; } //Exact 4 characters
    //and so on
}

... then I do all sort of data manipulation, queries, charts... you name it... everything using Linq.
We are testing a 1.8GB file and when it is opened, the process takes around 2GB of memory.
Eventually, when my customer needs to open his 10GB file it will be impossible, because it is going to take 12GB+ of Memory.
What is the best type/collection/list/dataset to this kind of work?

A: When i've had to do something like this before I handled it by having a container object that held a list of dictionaries.  At the time I thought the limit would/should be 2^32 number of elements, but an exception for exceeding the collection was thrown  well before getting 2^32 elements and still had many GB of ram left.   Say you want a Dictionary, something like this should work until you really do exhaust all physical and virtual memory...  A possible solution for you follows...  I remember when I worked on this a few years ago the server actually had 512Gb of ram, I'm sure they have ones with more now...  Anyway that's a separate story.
   public class MyHugeDictionary  
   {  
        List<Dictionary<typea, typeb> allDict= null;  
        Dictionary<typea, typeb> currDictionary ;  

        MyHugeDictiionary()  
        {  
            allDict = new List<Dictionary<typea, typeb>();  
            currDictionary = new Dictionary<typea, typeb);  
            allDict.Add(currDictionary);  
        }  

        public bool ItemExists( typea, typeb)  
        {  
            foreach( KeyValue<Dictionary<typea, typeb> kv in allDict)  
            {  
                if( kv.ContainsKey(typea) )  
                {  
                    return true;  
                }  
            }  
            return false;  
        }  

        public Add( typea a, typeb b)  
        {  
            try  
            {  
                if( !ItemExist( tyepa, typeb) )  // find if items is in any other dictionary first  
                {  
                    currDictionary.Add( a, b) ;  
                }  
                else  { // handle dups... ; }  
            }  
            catch( CollectionSizeError x)   // look-up for actual exception
            {  
                currDictionary = CreateDictiionary();  
                allDict.Add( currDictionary ) ;  
                currDictionary.Add( a,b);  
            }  
            catch( OutOfMemory y)     // look-up for actual exception
            {  
                // oops game over for real now :(  
            }  
         }  
    }  


A: After some discussion the best thing is to read the file, process it, and dispose all the rest, sticking only with the result.
Another possibility was to use database, but it would add too much complexity, although it is possible.

A: See this:
https://github.com/aumcode/nfx/tree/master/Source/NFX/ApplicationModel/Pile
https://www.infoq.com/articles/Big-Memory-Part-3
You can store whatever you want - no pauses.
The problem with large collections is:
a. They are not really designed to hold very many entries (i.e. Dictionary never shrinks back to zero size)
b. You get GC stalls/pauses when you have too many objects
see the links above - what we did is "hiding" of data from GC as described in the article. This way you can store millions of objects using LocalCache class as a dictionary.
For large memory apps in net - remember to enable 64 bit and set GC to SERVER mode in your app config file
