title: CSV Parse Taking Too Much Memory
body: <p>I'm trying to read a 5MM line file and right now it is exceeding my alotted memory usage on heroku. My method is somewhat fast ~200 inserts/second.. I believe it's crashing on the import.. So my plan was to import in batches of 1,000 or 10,000. My question is how do I tell that I'm at the end of my file, ruby has a <code>.eof</code> method but its a <code>File</code> method, and I'm not sure how to call it in my loop</p>
  
  <pre><code>    def self.import_parts_db(file)
          time = Benchmark.measure do
              Part.transaction do 
                  parts_db = []
                  CSV.parse(File.read(file), headers: true) do |row|
                      row_hash = row.to_hash
                      part = Part.new(
                          part_num: row_hash["part_num"], 
                          description: row_hash["description"], 
                          manufacturer: row_hash["manufacturer"],
                          model: row_hash["model"],
                          cage_code: row_hash["cage_code"],
                          nsn: row_hash["nsn"]
                          )
                      parts_db &lt;&lt; part
                  end
                  Part.import parts_db
              end
          end
          puts time
      end
  </code></pre>