In-Depth

The Rise of Solid-State Drives in the Datacenter

Solid-state drives (SSDs) have become more viable for boosting enterprise application performance and now the new Microsoft Windows Server 2012 R2 adds improved SSD support.

• By Jeffrey Schwartz
• 11/04/2013

When the Middle Tennessee Ear, Nose and Throat (ENT) Clinic stepped up its efforts to digitize all its patient information a few years ago, accessing the large files stored in its electronic medical records (EMR) system became painfully slow. The clinic's eight physicians and audiologists and other medical support staff were frustrated by the long time it took to pull data off the server's disk drives. To solve the problem, the clinic's IT organization added flash-based solid-state drives (SSDs) to its dual-core servers.

The clinic, with two locations in the Nashville metropolitan area, upgraded its Windows Server-based eClinicalWorks EMR system last year to the latest quad-core servers and a mixture of SSDs and traditional hard drives connected via a Serial Advanced Technology Attachment (SATA) interface. That helped boost performance. Like many enterprises of all sizes are finding, the limits of traditional mechanical disk drives have become a key bottleneck in system performance.

"It didn't matter how much horsepower, RAM or CPU processing we threw at it, it was an all-disk-based bottleneck," says Christopher Williams, the clinic's head of IT support. Installing SSDs and server caching software from FlashSoft Corp., which was acquired last year by flash storage giant SanDisk Corp., solved the problem, says Williams.

"We got an evaluation, used it for a few months and it made a huge difference," he says. "After two or three days, we intelligently started configuring things. After the first three or four months it was a no-brainer. We started using it live."

Hybrid combinations of flash drives and mechanical disks have become a viable option for organizations of all sizes. There are many ways IT can architect SSDs into the storage mix. Many are simply adding SSD drives to the PCIe slots in the servers, which also have hard disks (or configuring new server purchases with flash drives) or adding them as direct-attached storage (DAS). Companies managing storage networks are also increasingly adding flash to their storage arrays. In a growing number of shops where performance is critical, some are stepping up to pure flash-based SSD storage.

The Need for Speed
Disk drive capacity has grown exponentially in recent years, but disk drive speed hasn't increased, experts say. Today a typical server processor can read data out of main memory at 100 nanoseconds, while it takes about 500,000 times that long to read data off the disk, explains Howard Marks, chief scientist at DeepStorage LLC, a Santa Fe, N.M., consultancy and testing lab.

"Every time we need disk I/O, the system is incredibly slower," Marks said, speaking at a session on SSDs in the enterprise at last month's Interop trade show in New York. "Luckily, flash fills that gap. While a disk is half a million times slower than main memory, an SSD is only 1,000 times slower."

Another factor that's making flash-based SSDs an attractive option to a growing number of shops needing faster performance is that they continue to drop in price. Still, they're more expensive than hard disk drives (HDDs). As a result, a growing number of organizations are using hybrid SSD-HDD combi­nations with software to intelligently cache and put hot data on the much higher performing SSDs, leaving cold data on the hard drives. As hot data becomes cold or vice versa, the software moves the data to the appropriate media.

Hosting facilities, cloud services providers and many of the largest and most high-traffic sites are using flash storage to enhance performance, including Amazon.com Inc., Google Inc., Microsoft and even Facebook Inc. Many large banks, financial services and trading firms have little trouble affording or justifying the addition of flash storage to boost performance. At the same time, the media most commonly associated with the storage on smartphones, tablets and digital cameras is finding its way into mainstream IT shops.

These IT organizations might not be adding flash storage for all their applications, but certainly those with critical performance requirements. The Middle Tennessee ENT is now planning to expand its use of flash-based SSDs to serve up CT scans. A single scan can be as large as 1GB per image, with a typical set of tests consisting of up to eight scans, says Williams.

Only hot data is stored on the SSD drives, while the rest is stored on HDDs. Keeping it all on SSDs isn't necessary and wouldn't be economically feasible, says Williams. "To store it all on SSDs would be [unnecessary and] cost-prohibitive," he says.

Rapid Enterprise Flash Growth
Flash-based SSDs are certainly not new, but their presence in the datacenter is increasing at a rapid pace. Enterprise revenues for SSDs two years ago were $592 million. They're now on pace to top $3.5 billion by 2016, according to a report by Objective Analysis. That's a hefty 43 percent compounded annual growth rate.

The vastly improved performance of flash over disk comes from lower latency and the ability of SSDs to transact many times more Input/Output Operations Per Second (IOPS) than possible on disks, which are bound by the physics of the spinning mechanical drives. Most experts say flash storage performs 10 times faster than traditional hard drives found in servers and storage arrays.

Until recently, if you had a SQL Server application in an OLTP or business intelligence/big data analytics environment that needed more performance, you would add more drives or stripe volumes across more spindles. But given the smallest drives are now 300GB, you can end up adding terabytes of capacity via a slew of drives you don't need just to boost performance. Because many database and Web applications need performance but might not require tremendous capacity, flash drives have become more viable.

As the cost of flash technology comes down, it will become feasible to run more applications on SSDs, says Jay Prassl, vice president of marketing at SolidFire Inc., a startup that offers SSD arrays with software designed to optimize the use of enterprise applications using flash storage. "As costs come down and densities go up, you're going to see them used just about everywhere," he says.

That might be optimistic, according to Marks. While flash drives are coming down in price at a rate of 30 percent per year, already much less-expensive HDD prices are declining about 20 percent annually. "Hard disk drives will always be cheaper," Marks says. Nevertheless, Marks points to the growing use of flash in the datacenter, especially for Oracle and SQL Server. Experts say many are using it for a wide variety of other systems, including SharePoint, Exchange Server and Lync Server. Back-end flash storage is also suited for virtual desktop infrastructure (VDI) implementations, where the ability to boot from an SSD on a server can mean faster startup and access to data.

There are numerous other applications flash storage can enhance. "We're looking at using it with our content indexing, where we route a whole bunch of data through a server in order to scape every piece of data so we can search it at a text level, e-mail attachment and document level," says Jeff Echols, senior director of Business Development at CommVault Systems Inc., whose Simpana software is used for enterprise data protection and management. "That takes processing power, so being able to start implementing flash for that kind of process has become more feasible."

Just how much can flash storage improve performance over HDDs? The performance with even the fastest 15,000 RPM serial-attached SCSI (SAS) drives when wide-striping the data will average 200 IOPS. "If I've got an Oracle database that demands 20,000 or 30,000 IOPS, but is only 100[TB] or 200TB in capacity, I'm very quickly going to have a huge amount of capacity that's unused because those drives are only providing capacity," explains Marks. That's why using flash for even just 10 percent of your storage can address those performance bottlenecks, according to Marks and other experts.

Turning Flash to Gold
The growth of flash storage has caught the eyes of private equity investors and Wall Street alike. Pure Storage Inc. is another rapidly growing provider of an all-flash SSD storage array platform. Earlier in August, Pure Storage picked up a hefty $150 million in funding from a group of investors, including T. Rowe Price, Fidelity Investments and Tiger Global Management. The latest investment boosts the total funding in Pure Storage to $245 million, giving it a market cap of more than $1 billion.

Pure Storage launched its first product last year. The company claims it can offer flash-based storage in its arrays at a cheaper price than hard drives. "It depends on the deployment, but if you look at most people's tier 1 primary disk environments, they'll pay somewhere between $3 to $5 per gigabyte for usable higher-end disk," says Matt Kixmoeller, vice president of Product Management at Pure Storage. "When we first launched our product in 2012, our goal was to drive that price for all flash under that $5-per-gigabyte range. We were under that range last year and we continue to drive down the price this year."

Kixmoeller says his company has a wide array of customers from some of the largest Web infrastructure providers, financial services firms and arms of the federal government to high schools, hospitals and municipalities. Implementations range from 5TB to hundreds, he says.

While he acknowledges most aren't throwing away their existing disk arrays, "some of the more progressive customers we work with have moved to the point where they're now all-flash enterprises," he says.

While that might work in some instances, Marks warns of vendors claiming they can offer all flash at the same cost or cheaper than HDDs. "If you look at the systems from people who say 'I can deliver SSDs at the cost of a spinning disk,' they're not lying, they're just stretching the truth, because they're moving the cost," says Marks.

The business case for an all-flash array, says Marks, is if your applications consistently require 75,000 IOPS performance or more. And there's no shortage of companies -- both emerging and traditional -- jumping into the market. In addition to Pure Storage and SolidFire, relatively new players offering pure solid-state storage arrays include Kaminario, Nimbus Data Systems Inc. and Violin Memory Inc., which went public last month and announced last month it has worked with Microsoft to achieve 1 million IOPS with Windows Server 2012 R2 for scale-out cloud-centric infrastructure requirements.

Established infrastructure and storage providers are stepping up their investments in flash storage for the datacenter as well. IBM Corp. earlier this year said it was making a $1 billion investment in enterprise flash storage following its acquisition last year of Texas Memory Systems Inc. for undisclosed terms.

In September, Cisco Systems Inc. agreed to acquire Whiptail Systems for $415 million in cash. The networking giant claims Whiptail's solid-state memory system will bolster the Cisco Unified Computing System. Whiptail offers three lines of all-flash storage arrays ranging from 12TB to 360TB.

"As we continue to innovate our Unified Platform, Whiptail will help realize our vision of scalable persistent memory, which is integrated into the server, available as a fabric resource and managed as a globally shared pool," said Paul Perez, vice president and general manager of Cisco's Computing Systems Product Group, in a statement announcing the deal, which is set to close in the first quarter of next year.

Storage giant EMC Corp. has offered SSDs as an option for its VNX and Symmetrix arrays for several years, but made a stronger push with last year's acquisition of XStremIO. The company recently rolled out a new version of its VNX arrays with an upgraded version of the XStremIO software called EMC XtremSW Cache 2.0. The company says it improves latency by 65 percent. Marks notes that Hewlett-Packard Co., Dell Inc. and NetApp Inc. have all improved the software in their arrays to support SSDs in recent months.

Windows Server 2012 R2 Enhances Flash Microsoft's new Windows Server 2012 R2, released last month, offers a number of key new features, including a substantial upgrade to the Hyper-V hypervisor, support for live migrations and tighter integration with Windows Azure. Microsoft has also made the upgraded server OS more appealing to shops using flash-based SSDs with their servers thanks to the new caching and tiering capability in Storage Spaces. That's the technology in Windows that lets you create storage pools out of groups of drives and use that capacity to create virtual drives. "Microsoft has added a lot of enhancements to Windows Server 2012 R2 to make use of direct-access flash storage and PCIe flash," says Ray Lucchesi, a Microsoft MVP and president of Denver-based Silverton Consulting Inc., a consultancy specializing in systems and storage. "That should enhance usage of [SSDs]."

Lucchesi explains that Windows Server 2012 R2 automatic storage tiering allocates a portion of storage in the SSD to create tiers during configuration. It moves file segments to appropriate tiers to ensure adequate performance. Another feature in Windows Server 2012 R2 that improves SSD performance is Storage Spaces write-back cache, which, when enabled, writes to a space cached in the SSD and then allocated to an HDD when appropriate to improve performance.

With Storage Message Block 3.1 and Hyper-V 3.1, says Lucchesi, the Windows Server 2012 R2 storage performance is rated at 600,000 IOPS. "In order to do 600,000 IOPS in the old days would have required probably 2,000 disks, and enterprises-class storage systems with 16 interface and so on," he says. "With Windows Server 2012 R2, you can do it with a couple of SSDs -- maybe eight or nine -- and the performance is just amazing."

Shops using Windows Server 2008 and Windows Server 2012 can also find reason­able performance using SSDs, says Lucchesi. But the new Windows Server 2012 R2 release lets you tier between flash and disk drives without requiring third-party flash acceleration and caching software from the likes of Fusion-io Inc., LSI Corp., SanDisk and Stec Inc. (acquired in September by Western Digital Corp.), among others.

"It's sort of a replacement for those storage-tiering services that are out there," Lucchesi says, though in some instances they may offer advantages, especially when bundled with SSD media.

Marks agrees, saying while Windows Server 2012 R2 offers better flash performance than previous versions, many shops will choose the third-party options. "It's better-suited for flash, but you still need to do something with the throughput," Marks says of his preference for third-party tools added to Windows Server. "Curiously, Microsoft built what looks like a pretty nice write cache in Storage Spaces, but they didn't do any read caching. Write caching is more sophisticated and more complicated, but most of the benefit comes from read caching."

Rick Peterson, director of Software Marketing at SanDisk, says its FlashSoft caching software performs two functions. "First, it stands in the I/O path to identify what data blocks are being accessed most frequently and makes sure that those are available on the SSD on the cache. Second, is to make sure the cache is synchronized with the underlying storage," he says.

"Caching will drive better read performance while tiering provides both read/write performance improvements," says Blair Parkhill, vice president of marketing at X-IO Technologies, a provider of data storage software spun off from disk drive maker Seagate Technology LLC. "A common drawback to tiering is that movements are made between the media types in a matter of hours or days. Our technology called CADP gives us the benefit of moving hot data as quick as every five seconds, so the benefits are realized rapidly."

When Microsoft released Windows 8 and Windows Server 2012 last year, it introduced its Offload Data Transfer (ODX) interface, which is aimed at providing faster data transfer to storage devices, including everything from new media to cloud services. Microsoft says ODX reduces traffic and CPU utilization during large data transfers because all of the data transfer takes place in the back-end storage network. Designed for VM deployment, large data migrations and tiered storage, Microsoft says ODX reduces the cost of hardware with its thin-provisioning storage capability.

"The ODX APIs allow storage vendors like Pure to integrate with Hyper-V at much higher performance levels," says Pure Storage's Kixmoeller. "We are fully implementing those APIs in our file certification process with Microsoft."

Prior to the release of Windows Server 2012 R2, Redmond magazine Contributing Editor Brien Posey evaluated the beta and explained how to use the new OS storage-tiering feature for SSDs (see "Windows Server 2012 R2: Exploring Storage Tiering".

Posey warned it might not work on older hardware. "Microsoft said R2 supports two tiers, SSD and HDD," the company said in response to an inquiry about the hardware requirements for using storage tiering and SSD storage. "We require that the SSDs and HDDs can be identified accordingly and distinguished such that the tiers can be created properly. There are R2 logo requirements to ensure that SSDs and HDDs respond correctly to our identification logic."

As SSDs can boost the performance of virtualized environments, VMware Inc. has added support for enterprise flash storage in its recently released vSphere 5.5 virtualization platform. The company says its new vSphere Flash Read Cache allocates flash components at the per-VMDK level, offering consistent read caching and integration with the vSphere Distributed Resource Schedule for initial placement.

While it's noteworthy that VMware is adding flash support to vSphere, Marks describes the current iteration, also known as vFlash, as primitive. "You have to statically allocate flash space to individual guests," says Marks. "Not only is that a huge pain for the administrators, it also means that much flash belongs to each of those guests, so through the day, as this guest gets busy, it doesn't get more flash, and then as the other one gets busy the flash doesn't get dynamically allocated."

SSD Limitations
Flash might improve your application performance, but not all flash media is the same. The most common types of flash are single-level cell (SLC) and multi-level cell (MLC). The latter is more commonly found in inexpensive laptops, tablets and other consumer devices, though enterprise-grade MLC is also available.

The quality is based on the NAND in the memory. In general, MLC flash, which stores data in multiple memory cells, will only allow between 5,000 and 30,000 program-erase cycles before the media is no longer reliable, says X-IO's Parkhill. The more reliable SLC, which stores data in individual memory cells, is found in enterprise storage systems because it allows 100,000 program-erase cycles before wearing out, he says.

"The problem with SLC is it costs 10 times as much as MLC because it's only used by enterprise customers and SLC's only make up 12 percent of flash consumption," Marks adds. Whichever form of flash media you choose, it isn't recommended if indefinite data retention or archiving is an objective or requirement. SSDs are bad long-term storage media. "SSDs are not a place you want to keep data for 10 years as the charge will dissipate and you will lose data.