I'm currently tasked with the difficult problem of figuring out how to efficiently pack an image and some text within a single file. In doing so, I need to make the file relatively small (it shouldn't be much bigger than the size of the image file alone), and the process of accessing and saving the information should be relatively fast.
Now, I have already found one way that works - converting the image to a string using pygame, storing it (and the text I need) within an python object, and then pickling the object. This works fine, but the file ends up being much MUCH larger than the image, since it's not being compressed. So to help with this, I then take the pickled object and compress it using gzip. Now I have another problem - the whole process is just a tad bit too slow, since I'll need to do hundreds of these files at a time, which can take several minutes (it shouldn't take longer than a 1/2 second to load a single file, and this method takes up to 2 seconds per file).
I had an idea to somehow put the two separate files, as they are, into one file like how someone would with a .zip, but without the need to further compress the data. As long as the image remains in it's original, compressed format (in this case, .png), simply storing it's data with some text should theoretically be both fast and wouldn't use much more memory. The problem is, I don't know how I would go about doing this.
Any ideas?
Alex Reynolds idea to use tar archives seems to be a perfect match.
How would I go about serializing multiple file objects into one file?