Any suggestions how I can distribute millions of files from 1 server to X number of other servers? I'm looking more into an algorithm on how to decide which server to send the file to.:
Perhaps look at a distributed filesystem like GlusterFS. It sounds like it will meet all your requirements and will probably be more reliable than something that you hack up yourself.
Despite your impossible requirements, I'll scribble down my thoughts for other people in the future who aren't so hamstrung, based on my experiences doing this for Github.
Distributing data across a number of locations (be they partitions, machines, data centres) based on a hash is a dangerous undertaking, for two reasons:
On the other hand, having a lookup table for all your files makes these problems go away. When you say "no database", I'm betting you insert an implicit "SQL" before "database". However, there is a whole other world of databases out there that have nothing to do with SQL, and they are perfect for this situation. They're known as "key-value stores", and if you're dead keen on going ahead with building this boondoggle yourself, then I'd highly recommend using one (I've got experience with Redis, but they all seem pretty reasonable).
Ultimately, though, if you go ahead with the "all hashes, all the time" system and then hack around the problems inherent in it (there are solutions, just not real awesome ones) all you will end up with, at the end of the day, is a half-assed, botchy, non-feature-complete version of GlusterFS. If you need a large amount of storage, growable over time, distributed across multiple physical machines, in a single namespace, I really would recommend it over anything you can build yourself.
If you still want to hack it, do a md5sum on each file and then hash the output to your X boxes..
If you have two boxes:
0*-7* go to box one
8*-f* go to box two...
Or if you have 256 boxes:
00*-0f* go to box one
10*-1f* go to box two..
and so on..
This works best for box counts of powers of two.(2,4,8,16,..)
Keep in mind that shuffling things off is all nice and good, but you'll want
to keep an index somewhere if you also need to retrieve this info.
(where did I put foo.txt??)
A flat file pickle (in python) would work, but it won't scale as well as a DB for large amounts of data..
Can the other servers also send files? Are you in a "safe" environment?
The Rocks clusters installation process has to fill rack after rack of compute nodes, each one installed on the fly from an initial image. Doing that linearly or through a single server would be a bottleneck. Rocks uses instead a little system called Avalanche, where the install images are served using p2p; as nodes come up, they also become servers that will be used to install new nodes. The result is a tree of servers, and the install images cascade through the racks very quickly. The overall latency is a logarithm of the number of nodes, multiplied by the time to install one node (the base for the logarithm depends on how many other nodes can be served from one that is already installed, log base 20 wouldn't be surprising...).
You could imagine a similar strategy for copying out your files, but only if the destination servers would be willing to trust other servers for their copy.
asked
5 years ago
viewed
216 times
active
2 months ago