title: python pickle vs sql efficiency
body: <p>I'm developing an application in Python which requires storing (very) large data sets. Is pickle the most practical way to store the data and retrieve it on request, or should i consider using SQL instead? My main goals are speed and as little processing strain as possible.</p>
  
  <p>My concern is that pickle has to process an entire large file on the fly, which could adversely effect performance. I'm not particularly familiar with pickle beyond usage, so any explanation to how it works would be great.</p>
  
  <p>Right now, i'm using this code:</p>
  
  <pre><code>users = pickle.load( open( "users.py", "rb" ) )
  username = raw_input("Please enter a username: ")
  password = raw_input("Please enter a password: ")
  var = username in users
  if(var == 0):
      return 0
      exit()
  else:
      if(users[username] != password):
          return 0
          exit()
      else:
          return 1
          exit()
  </code></pre>
  
  <p>Imaging that users contains 1 million entries, which would be more efficient, this or SQL?</p>
  
  <p>Any help would be great,</p>
  
  <p>Thanks</p>