title: Python - splitting up many small txt files into a data structure to feed into mysql
body: <p>I have a few hundred thousand txt files that are a pretty standard shape (they all have common elements - ID, Date, To, From, Subject, Body. </p>
  
  <p>These are not in a structured format e.g. a multipart email message. </p>
  
  <p>I want to strip these into their constitute parts, and feed the whole lot into a db. There are lots of them, so I wanted to make sure the approach would work. </p>
  
  <p>There are a number of key issues I am pondering (and I am no coder - this is learning / hobby stuff). </p>
  
  <p>1) is there a structure data type I can cast to keep the bits together in a sensible way. I was thinking it would be logical to have a file.ID, file.Date, etc tryp deal that holds the whole file in a structured way so it can be later ingested into the db. Is this pythony? or a hangover from my tinkering with Matlab?</p>
  
  <p>2) the body section can be several kb large or a single sentance. (1) is this better as a blob - I would loose the searching - kind of the point of doing this.. and (2) How do make sure I can construct a field large enough in my MYSQL database come ingest time? I won't know what the longest size is of each of the element unless I run some kind of counter in the message splitter that handles the max value seen per message</p>
  
  <p>3) I figure that I would start with a walk, get the file list from the walk, then pull each file, line by line. I'll use line position to infer some known locations (ID, Date) and then some RegEx or patterns based on features to split the rest. Once I have split the files up, I plan to ingest them. However, I wonder if it would be more logical to connect to the db at the end of each message and dump the parts into its own record one by one. </p>
  
  <p>Time is no drama, it can run for a week for all that matters. I have about 8gb of RAM on an i7, so again I'm not resource hungry specifically, and happy to let it grind its way through. </p>
  
  <p>Does this sound logical? Have I missed a core step?</p>
  
  <p>Thanks. </p>