(Edited: My bad, it was 1.3TB/s, not 1TB/s). What do you do when
you need so much I/O performance that no one single storage
system can deliver it, no matter how large? To be specific: What
if you needed to transfer data at 1TB per second? (or 1.3TB/s,
as it eventually turned out to be)? That was the problem faced
by the U.S. Department of Energy (DoE) and their Sequoia
supercomputer at the Lawrence Livermore National Laboratory
(LLNL), one of the fastest supercomputing systems on the planet.
You can read the official press release here. I wanted to get
more into the technical details. People talk a lot about “big
data” recently – no clear definition seems to exist, in my
opinion it’s something that has some of the following
properties: - Too much data to be processed by a “normal”
computer or cluster - Too much data to work with using a
relational DB - Too much data to fit in a single storage system
for performance and/or capacity reasons – or maybe just simply:
- Too much data to process using traditional m