We are using Lustre on a projec.  The differences that jump out to me on <a href="http://ceph.newdream.net/about/" rel="nofollow">http://ceph.newdream.net/about/</a><p>1) auto-balancing when new storage nodes are added<p>2) copies of objects stored in multiple storage nodes<p>Running HA Lustre requires a LOT of work.<p>1) Lustre requires tricks like copying files to rebalance<p>2) Lustre only has one copy of any section, so each OST is a SPOF.  Sun recommends deploying OST in pairs with DRDB and Heartbeat functionality (cutting your space in half and complicating the deploying), but a box failing won't break your FS.  This doesn't fix the problem that if a network partition occurs since you still have locality of chunks in a single rack.<p>For more information about how complicated Lustre in a production environment is check out the talks at the last UG by Sun employees (Lessons Learned &#38; Best Practices, Managing High Availability on a Shine-Equipped Lustre Cluster):<p><pre><code>    http://wiki.lustre.org/index.php/Lustre_User_Group_2009
 </code></pre>
I've not tested Ceph in production but many people I talk to about our Lustre woes recommend Ethan Miller's work - <a href="http://users.soe.ucsc.edu/~elm/" rel="nofollow">http://users.soe.ucsc.edu/~elm/</a> - which leads to Ceph ;)<p>Even if you aren't worrying about availability, Lustre has issues with usage:<p>Lustre requires older kernels - which means compatibility with modern non-RHEL distros leads to headaches when you need the lastest version of KVM (or python).<p>Lustre also tries to be a POSIX filesystem but it breaks horrible when you try to use features like O_APPEND from multiple nodes (file corruption!)  We are still tracking down Lustre breaking when we read files in a certain order (rsync is ok, but start skipping around in a file read leads to the kernel stopping responding)<p>After months of dealing with Lustre issues with trying to support POSIX I have come to love the non-POSIX (S3-like API) to distributed filesystems.