Question: I recently had a couple of 10Gbit Ethernet cards installed on a few machines connected to a LAN of about 80 commodity nodes running a distributed file system (Lustre). The 10Gbit card achieves good performance on file operations and is functioning as it should.

However, I wrote a custom client app in C that asynchronously sends large blocks of data to multiple nodes in the network. The machine the client app runs on has a 10GB Ethernet card, and all the destination nodes have a 1GB Ethernet card - so I should be able to get a theoretical max send transfer rate of 10 gbits. 

If I run the client app on a machine with a 1Gbit card, it easily maxes out the card for sustained lengths of time. But strangely, if I run the same app on a machine with a 10Gbit card it gets horrible performance (around 20-30 mbits a second).

The program is written in C using normal TCP sockets. Is there some special setting required for 10Gbit? Because it's very odd that it gets maximum performance on a 1Gbit card, but horrible performance on a 10Gbit card. Again, the problem is not the 10Gbit card itself, since the distributed file system (Lustre) gets good performance with the 10Gbit card.

Any ideas/suggestions?

Answer: 
One thing I've noticed as a problem between 10gb and 1gb lan segments is that the default MTU is different. 10gb ethernet uses a default MTU of 9000 as opposed to the default of 1500 for 1gb ethernet. You can either change your MTU on the 10g to the lower number or set up your router to handle breaking down the jumbo packets for you. 

This has caused me some headaches, because without one of those two things configured, there is a lot of packet fragmentation.
It is very likely that the application you wrote is not at par with the I/O optimizations of a standardized applications like Lustre.

The performance bottle necks in your code may not be surfacing on the machine and OS with a 1Gbps card but when the card throughput capacity increases to 10Gbps, with all other parameters constant (hardware and OS), your code limitations are made prominent.

---

This is quoted from the Wikipedia [Lustre Implementation](http://en.wikipedia.org/wiki/Lustre_(file_system)#Implementation) section.

> 
> In a typical Lustre installation on a Linux client, a Lustre filesystem driver module is loaded into the kernel and the filesystem is mounted like any other local or network filesystem. Client applications see a single, unified filesystem even though it may be composed of tens to thousands of individual servers and MDT/OST filesystems.
> 
> 
> On some massively parallel processor (MPP) installations, computational processors can access a Lustre file system by redirecting their I/O requests to a dedicated I/O node configured as a Lustre client. This approach was used in the LLNL Blue Gene installation
> 
> 
> 

Are you using this part?

> 
> Another approach uses the liblustre library to provide userspace applications with direct filesystem access.
> 
> 
> Liblustre allows data movement directly between application space and the Lustre OSSs without requiring an intervening data copy through the kernel, thus providing low latency, high bandwidth access from computational processors to the Lustre file system directly.
> 
> 
>