A Pragmatist’s view of PCIe SSD

Yesterday TMS announced our latest PCIe Flash offering, the RamSan-70.  Prior to the launch, I had the chance to brief a number of analysts and to explain the key deployments we are tackling with this offering.  One thing I discovered was that there is a lot of confusion about what PCIe SSDs bring to the table and what they don’t.  So with that in mind, I present the following primer on fit of PCIe SSDs.

How would you describe a PCIe SSD?

A PCIe SSD is like Direct-attached storage (DAS) on steroids.

Doesn’t being on the PCIe bus increase performance by being as close to the CPU as possible?

Yes, but nowhere near the degree it is promoted.  Going through a HBA to FC attached RamSan adds about 10 µs of latency –that’s it.  The reason that accessing SSDs through most SAN systems take 1-2 ms is because of the software stack in the SAN head – not because of the PCIe to FC conversion.  For our customers the decision to go with a PCIe RamSan-70 for a FC/IB attached RamSan-630 comes down to whether the architecture needs to share storage.

Are you working on a way to make the PCIe card sharable

No, we have shared systems.  If the architecture needs the shared storage, use our shared storage systems.

So why is PCIe making such a splash?  Isn’t the DAS vs SAN argument over with SAN rising triumphant?

Well, the argument was over until two things happened: servers started to get really cheap, and really, really big clusters started getting deployed.  In a shared storage model, a big core network is needed so each server can access the storage at a reasonable rate.  This is one of the main reasons a dedicated high performance Storage Area Network is used for the server to storage network.  However, after there are more than a few dozen servers, the network starts to become rather large.  Now imagine if you want to have tens of thousands of servers, the network becomes the dominant cost (see the aside in my post on SSDs and the Cloud for more details).  In these very large clusters the use of a network attached shared storage model becomes impractical.

A new computing model developed for these environments – a shared nothing scale-out cluster.   The basic idea is that each computer processes a part of the data that is stored locally, many nodes do this in parallel, and then an aggregation step compiles the results. This way all of the heavy data to CPU movement takes place within a single server and only the results are compiled across the network.  This is the foundation of Hadoop as well as several data warehouse appliances.  In effect, rather than virtualized servers, a big network, and virtualized storage via a SAN or NAS array; the servers and storage are virtualized in a single step using hardware that has CPU resources and Direct-Attached Storage.

PCIe SSDs are important for this compute framework because reasonably priced servers are really quite powerful and can leverage quite a bit of storage performance.   With the RamSan-70 each PCIe slot can provide 2 GB/s of throughput while fitting directly inside the server.  This much local performance allows building high performance nodes for a scale-out shared-nothing cluster that balances the CPU and storage resources.  Otherwise, a large number of disks would be needed for each node or the nodes would have to scale to a lower CPU power than is readily available from mainstream servers.  Both of these other options have negative power and space qualities that make them less desirable.

The rise of SSDs has provided a quantum leap in storage price-performance at a reasonable cost for capacity as new compute frameworks are moving into mainstream applications.  Both of these developments are still being digested by the IT community.  You can see the big vendors jostling for dominance to control the new platforms that are used to build the datacenters of the future.  At TMS we see the need for “DAS on steroids” in the new frameworks and are leveraging our hardware engineering expertise to make the best DAS solution.