Q:

Efficient way to save data

In my simulation program, I calculate about 50 numbers in one step, and repeat this step about 100000 times. I need to save all these numbers for plotting them.
I use python,and I thought two ways.

do all calculation at once by appending these numbers to list  or
for each step, write 50 numbers in file, and repeat this.

I actually made this program by using the first way, such as
for i in range(100000):
    data.append(calculated numbers)

but I think this is too memory wasting.
but I also think that the second way is slower than first(?).
In this case, what can be efficient way?

A:

You will want to compute the numbers in batches and write them to disk as you go.  Depending on the file system you are using, you will want to write larger than a certain chunk size to avoid excessive disk writes.  Your performance hit in method 2 should be because you are triggering 100000 disk writes.  Instead, write to the disk after every 1000 cycles.  Then you only have 100 disk writes, but you only use about 50,000 numbers of memory at a time.  Then you have high performance and stay within your RAM limit.
Also, consider using xrange instead of range so you aren't allocating a list of 100000 numbers just to do the loop.

