ENTERPRISE STORAGE APPLICATIONS AND THE FLASH REVOLUTION

Flash memory can be used in many different ways to solve real-world issues in the enterprise. This white paper examines the various options available today and in the near future with a focus on use cases for enterprise storage requirements and application acceleration.

Will Alger January 2012

Executive Summary
The truth today for many enterprise IT organizations is that they're still adding physical capacity to disk arrays that have long since run out of "I/O capacity." For many applications, I/O performance is far more important than total capacity. The practice of meeting a specific I/O objective by adding disk spindles to a storage subsystem has many potential pitfalls; excessive power and cooling requirements a large physical rack space required to house a sufficient number of disk drives to meet the performance need an increase in the potential for drive failure as the result of having an extended quantity of disk drives high acquisition costs without necessarily solving the problem at hand additional effort is required to effect optimal data placement for I/O performance a majority of the disk capacity can go unused in order to obtain the I/O objective

There have been many different approaches to providing solutions for these issues. A significant change in the storage industry has been driven by the increased use of SSD (Solid State Disk), also sometimes described as EFD (Enterprise Flash Disk) but in either case based on flash memory technology. For the purposes of this discussion we will refer to SSDs in the context of flash, but generically the term SSD can refer to other technologies such as those based on RAM memory. Flash memory has the potential to provide a number of benefits to resolving the trade-off between I/O and capacity that is a current challenge for many storage deployments. Flash provides extremely fast random I/O in comparison with standard disks, so its most obvious applicability is in the areas where small-block random I/O is impacted. These include databases, OLTP, Virtualization (hypervisorbased) environments and VDI (virtual desktop infrastructure). The intent of this paper is to examine some of the different ways that enterprise storage suppliers are incorporating flash into existing or new products, the advantages and limitations of the different options and some of the upcoming developments in this exciting area. The underlying focus will be to examine the ways in which the products described can be used in enterprise storage deployments and to provide some guidance in selecting the best solutions for specific requirements. What this paper does not address is the SSD market as a whole; the focus is on enterprise requirements that are characterized by a given products applicability to an enterprises needs rather than those of a consumer or a small / medium business. Some of the criteria that can apply to products intended for enterprise class use include extremely high levels of availability, non-disruptive upgrade capability, scalability in terms of performance and connectivity, and a comprehensive support program to ensure continuous access to and utilization of the products features.

Will Alger

January 2012 1|Page

Brief Background
The concept of using some form of semiconductor-based memory as a replacement for spinning disk drives or to augment the performance of a disk subsystem has been around for quite some time. We may think of this technology as being relatively new, however it has existed in one form or another for several decades. Products using DRAM or SRAM to mimic the functions of a disk drive have been in use since the 1970s. At one time bubble memory was seen as a legitimate competitor to disk drives. Companies such as Texas Memory Systems (TMS), Dataram and Imperial Technology were early innovators in this market. The high cost and low-density of the early generation products limited their deployment to relatively niche applications. Many factors have reinforced the dominance of disk drives as the mainstay of primary storage over time. Data stored on disk is non-volatile; it does not require that a disk be powered on to be retained. Development of new technologies that increase the areal density on disk platters has allowed for huge amounts of capacity to be provided. Disks provide relatively good performance and are a well-characterized building block that has plenty of standardization so multiple sources of supply can be interchanged relatively easily. Above all, disk drives are extremely cost effective in comparison with other types of storage devices and fulfill their role fairly well for many different applications. Disk drives have some negative characteristics however; since they are electromechanical devices they inevitably wear out. Over time, the capacity increase achieved with disks has not been matched with a corollary increase in performance so storage design with disks alone has not always kept pace with application requirements. Disk-based storage subsystems can be heavy consumers of power and cooling, a factor of concern in modern data centers. And finally, in most current disk subsystems there is a discrepancy between maximum capacity and maximum performance; very few if any can be said to provide 100% utilization, which would be characterized as being able to maintain a linear relationship between capacity and I/O. Disks at a device level and a storage subsystem level have incorporated cache to accelerate some aspects of performance. Caching helps, however it does not have universal benefit for all I/O workloads and at a subsystem level is an expensive proposition. An interesting application in some storage subsystems is the ability to treat a portion of the cache memory as a virtual disk; this can be a challenge as far as subsystem tuning since the isolation of part of the cache detracts from overall storage performance. Nevertheless, the concept of cache as virtual disk has some applicability for uses such as providing extremely high performance repositories for database re-do logs, as one example. This concept has been expanded to utilize SSDs instead of SDRAM for a similar purpose in some implementations.

Will Alger

January 2012 2|Page

Flash Forward to Today

The market today for SSDs and flash-based products of all kinds has been greatly expanded by the advent of lower cost and higher density packaging. Flash has the potential to become the dominant storage medium for many types of enterprise applications. A number of different packaging and implementation approaches have been used to support flash memory-based storage. These include: PCI-Express cards Disk drive replacements in a storage subsystem Caching appliances Standalone storage devices using existing storage interconnect methods Hybrid combinations of flash, disk and software

All of these methods can potentially provide benefits to application performance, however there are some inherent limitations to flash memory itself that have to be addressed. These include finite lifespans that are limited to an average number of writes per cell of the flash device(s). The types of flash devices in general use today include SLC (single-level cell) and MLC (multi-level cell). SLC-based devices are faster, have greater longevity (100,000 writes per cell on average vs. 10,000 for MLC / 30,000 for E-[enterprise]MLC) but are more expensive and provide less density than MLC-based devices. The flash market today has some interesting dynamics, to say the least. Traditional storage manufacturers face several challenges in how they address the incorporation of flash technology in their existing products; this is because their disk array architectures are the result of many years of development that at this point are primarily oriented to managing and maximizing disk performance. Because of this it is difficult to integrate SSDs in a manner that optimizes their use in existing products. This has created a huge opportunity for smaller companies to start with a clean slate approach to the development of interesting flash-based architectures because they have no legacy product portfolio to protect. The flash market is populated with literally hundreds of start-ups with interesting concepts but little breadth in their product lines. Are these suppliers viable for enterprise applications? We need to apply the criteria of what constitutes an enterprise level solution to the various approaches and see if they really can deliver. Lets take a look at some of the representative product offerings in each of the categories we identified above.

Will Alger

January 2012 3|Page

Playing the Cards

Fusion-IO is probably the best known developer of PCI-E-based flash board level products. Founded in 2006, Fusion-IO became a public company in 2011. It partners with many of the best-known system suppliers such as HP, IBM, Dell and SuperMicro. Fusion-IOs products include the SLC-based ioDrive series, the MLC-based ioDrive, SLC and MLC variations called the ioDrive Duo and the super high-density MLC-based ioDrive Octal cards which can provide up to 10TB of capacity. One thing that I believe it is important to note about characterizing Fusion-IOs product performance claims (and this goes for all products, not only those from Fusion-IO), is a tendency to describe maximum theoretical numbers rather than demonstrating what can be obtained in real-world applications. The claimed million IOPS that Fusion-IO states for the 5TB ioDrive Octal, for example, is based on block sizes and parameters that would never be seen in real applications. If we were to look at a typical usage model for this type of product using 4K blocks, the actual IOPS achievable would be around 150K IOPS. That is still extremely good performance mind you, but the point is that your mileage will vary and it will not be the same as a maximum theoretical number. Do board-level flash storage products make sense? There are pros and cons to this approach. On the positive side, placing the devices in a PCI-Express backplane means that in theory the fastest possible performance can be achieved because there is no intervening connection layer. The limit to bandwidth and IOPS is whatever can be delivered by the PCI-Express architecture, which can be substantial- PCI-Express 2.0 x16 bandwidth is 8.0 GB per second before error-correction; with error-correction that number is about 6.08 GB/sec. This is a lot of bandwidth. PCI-Express specifications have been developed for versions 3.0 and 4.0, however there are no current implementations of storage devices on those. 4.0 specs are in a draft form at this point and not finalized. Another area to keep an eye on is the development of interconnect specifications that are geared to the high performance capabilities of flash-based systems; these include the NVMe (Non-Volatile Memory express) consortium which includes many of the major players in the flash market, as well as SCSI express, a method for allowing SCSI initiators and targets to communicate across the PCI bus. Thus far, SCSI express has been primarily touted and demonstrated by Fusion-IO in cooperation with HP. Despite the potential performance advantages of board-level storage devices there are a number of limitations to consider. Applications may need to be redirected to the internal device which can present difficulties. Because they are installed in an individual server, the capacity cant be shared with other host systems. The single-server approach has other disadvantages; backups can be problematic, and single points of failure can result.

Will Alger

January 2012 4|Page

Interestingly, other companies have incorporated Fusion-IO products into their own offerings. Some of these include Kaminario, combining their high-speed DRAM-based K2 product with Fusion-IO cards to create hybrid offerings, as well as Nutanix, a start-up company that has developed a three-tier storage device for VMware environments that includes the MLC version of the Fusion-IO ioDrive in addition to Intel SSDs and SATA disk drives. Fusion-IO is of course not the only company providing products of this type. Additional suppliers include OCZ, with some interesting board-level technology, TMS, a longtime supplier of storage-oriented devices but a relatively new player in the board market with what appear to be some extremely high-performance products, the memory manufacturer Micron Technology, and specialty provider Virident who focus on SLC-based PCIe cards. EMC, the largest supplier of storage products, announced its intent to introduce PCI-e flash products in 2012. It will be interesting to see what approach EMC takes in this market given that there have been a number of discussions about how, for example, EMC might use its VMAX storage platform to run applications under VMware, the implications of its RAPID-IO high speed interconnect technology in the VMAX and the possibility of cooperative processing between EMC storage subsystems and server-resident flash cards to cache applications.

Will Alger

January 2012 5|Page

SSDs in Existing Storage Subsystems

EMC created a great deal of press and expressions of skepticism from its competitors in January 2008 when it announced new product features for its high-end DMX storage subsystem that included new SSD drives. Using fibre channel SSDs sourced from STEC, EMC initially introduced 146GB SSDs of which a maximum of 128 could be installed in a DMX subsystem. At the same time EMC announced that it was developing an automated tiering facility to transparently locate LUNs on the most appropriate storage tier in a DMX, including the SSD layer which it called Tier Zero. Over the next couple of years this capability evolved into what is now called FAST, or fully-automated storage tiering, and which now provides sub-LUN or block level relocation over tiers. Its interesting that now, four years later, we can look back on that announcement from a very different perspective. The general consensus at the time from industry analysts and EMCs competitors was that the incorporation of SSDs was far too expensive for the majority of customers, which SSDs had limited utility for normal data I/O requirements and that no one would be willing to pay the prices that EMC was asking for its new SSD devices. Despite the naysayers EMC shipped over 10PB of SSD capacity in 2010, and indicated that in 2011 that number would be close to 30PB. In those four years, the majority if not all of the major storage vendors have also introduced SSD drive capability in their subsystems, so from that point of view it appears that EMC was being quite visionary at the time. Whatever the case, the need to catch up with EMC was certainly good news for STEC and other suppliers of SSDs with industry-standard interfaces that could easily be integrated into existing storage architectures. If we consider that the best rationale for incorporating SSDs into a disk subsystem is to be able to respond to I/O workload changes, which is to say being able to move blocks around to the most appropriate layer of performance, then some more investigation is needed. In general terms, the type of tiering solutions that have been developed by 3Par (now part of HP), Compellent (now part of Dell), Hitachi Data Systems and many other storage suppliers are very good at moving data constructs down the different layers but not really as efficient as moving them up. The original thinking about tiers at all was something like this; most data has a finite lifecycle, so as it ages it is accessed less frequently and therefore it should be moved to a more inexpensive storage medium. The paradigm for this approach was to use fast fibre channel drives as a Tier 1 and then as data aged move it to less expensive and slower SATA drives. In this scenario, it doesnt really matter how quickly the data is moved; it could take an hour, or a day, without really impacting performance (unless of course we have run out of Tier 1 capacity, which is a different discussion). However, consider the opposite condition, where we need to promote data volumes or blocks up to a faster tier to respond to a change in the importance of that same data; now it really does matter how quickly it can be moved. The reality is that the time required to examine and relocate data in this manner is not what storage tiering was really intended to do and that the existing implementations do not promote data as quickly as we would like to meet this requirement.

Will Alger

January 2012 6|Page

So why hasnt EMC, or HP or IBM or any of the big existing storage providers come out with an entirely new flash-based storage architecture that really addresses this issue? Its a reasonable question but there is no simple answer. In some instances it could be that such a move would tend to cannibalize existing revenues. It could be that the engineering expertise needed to create entirely new storage architectures is not available to those companies. Perhaps the big players are waiting to see what the market really wants based on the offerings available from the dozens of start-ups and smaller companies developing all-flash or hybrid storage systems and then to develop products of a similar scope or acquire the smaller players. SSDs are now available with fibre channel, SAS (serial attached SCSI) and SATA interfaces, so they can easily be incorporated into existing storage subsystems that use those same interconnects. Thats the good news; the bad news is that the use of those disk-oriented interfaces limits the potential transfer rate and I/O performance of the flash device. That is the message of the PCI-e flash manufacturers and it is a valid point. It is an extremely good idea to take a look at the entire scope of the applications that you want to accelerate if that is your goal, and see if there are more creative ways to address those requirements rather than simply adding SSDs to an existing storage platform which may limit the value of the entire exercise.

Will Alger

January 2012 7|Page

Caching Appliances
The basic concept of using a caching front-end to accelerate I/O requests is simple enough; after all, its what weve been doing in our storage subsystems for a long time. The potential appeal of a flash-based caching approach is that whereas most cache architectures in storage subsystems use expensive DRAM or SRAM in relatively small amounts, a flash approach could provide significantly more cache capacity and potentially accelerate workloads transparently and without changes to our underlying storage and fabric architectures. Another possibility to consider is that a cache appliance of this sort could also be used to sit in front of more than one storage system or heterogeneous mixes of subsystems and would thereby maximize and extend the investment in those systems. Products in this category follow the traditional delineation of SAN (block level) or NAS (file level) interconnects and data handling paradigms, which makes perfect sense. There have been some failed attempts at creating products of this type and a couple of companies working in this area have already come and gone, but there are some credible players as well. Perhaps the best known products in the NAS caching / acceleration market are those from Avere Systems. Avere manufactures the FXT series of appliances that incorporate DRAM, NVRAM and flash SSDs in different model configurations in conjunction with 10Gbit and 1Gbit Ethernet ports to provide caching for NAS storage subsystems such as those from NetApp. The FXT modules can be combined to create clusters of up to 50 modules, so they are very scalable. In the SAN market, there are similar products from Dataram, the XcelaSAN series, as well as from Gridiron Systems with what it markets as the Turbocharger. Conceptually these are very similar; the caching focuses on servicing read I/O and writes are passed through to the disk back-end. This makes good use of what flash is good at, random reads with a relatively low locality of reference and bypasses the issue of write integrity by committing writes directly to disk. Generalized benefit can certainly be obtained from caching appliances given appropriate workloads and expectations. One thing to consider is that these are single purpose devices; they cannot be redeployed to address changing requirements or conditions as could a standalone flash-based storage device. As a result it would be wise to compare the TCO and ROI metrics that can be accomplished with a caching approach vs. other possible solutions.

Will Alger

January 2012 8|Page

The All-Flash Array

In some ways the all-flash array is the most obvious direction for the effective utilization of flash memory in storage applications. If we look at it from the point of view of the number of companies working in this area it is where the highest level of activity currently is directed. Again, the basic concept is easy to grasp and familiar; use existing storage and fabric connection methodologies to provide a shared storage device based on highspeed flash. The most common product approach has been to use a fibre channel frontend; however there are also products that use Ethernet-based iSCSI or Infiniband. The best known company in this segment is perhaps Violin Memory; it is arguably the largest with revenues projected at approximately $100M in 2011. A large part of this success has come through partnerships with system suppliers HP and IBM, and more recently with Oracle. Violin was founded in 2005, and provides all-flash subsystems based on MLC and SLC technologies. Violin sells a NAS caching product as well, but they are primarily focused on the fibre channel networked storage market. Violin has addressed many of the product features that are necessary and desirable for its intended market, which is clearly oriented to enterprise applications. It provides a San Gateway that can aggregate multiple 3U memory arrays and provides connectivity to multiple hosts. In the areas of data protection and device integrity it has hot-swappable RAID protected VIMMs (Violin flash modules). From a software perspective it provides thin-provisioning for virtual volumes, CDP (continuous data protection) and snapshots in the array. Software will be a significant differentiation point in this segment of the market. In addition to the standard storage subsystem capabilities referenced above, all-flash array manufacturers have included advanced storage capabilities such as deduplication and compression to maximize utilization. One company applying many of these features is Pure Storage, with its dual-controller-based FlashArray. We will see more and more companies entering the pure play flash array market. A few others that have interesting products in this area include Nimbus Data with multiple interconnection options and SolidFire with an Ethernet-based product that is intended to appeal to Cloud storage providers. Flash storage arrays can use more bandwidth than is currently generally provided. Today, the most common front-end arrangement is to use 8Gbit fibre channel or 10Gbit Ethernet (and/or multiple 1Gbit Ethernet) ports into the array. Immediately around the corner will be the ability to use 16Gbit fibre channel and potentially 40Gbit Ethernet for this purpose. The increased bandwidth will make the flash-only array even more compelling for enterprise deployment.

Will Alger

January 2012 9|Page

Hybrid / ASAP Technologies

As discussed in the Existing Subsystems section above, one of the capabilities that would be highly desirable for incorporating flash into enterprise storage environments would be to tier between layers of performance in real-time. We talked about the role of traditional storage tiers, which do a good job of demoting data as it becomes stale but are not as efficient at elevating data blocks or files as they become hot. That is the province of what we might call hybrid storage devices, or as the editor of storagesearch.com has termed it SSD Acceleration as soon as possible, or ASAP. The basic idea is to integrate a high-speed flash tier for performance with a disk layer for capacity and be able to move blocks between the tiers extremely quickly to respond to changing I/O request rates. It is easy to see the desirability of this method; in a typical work day the storage workload is highly variable so A) it doesnt necessarily make sense to put all data on the fastest tier, and B) it would be great if the data was in the fastest tier at the right time. Thats the hybrid concept. Nimble Storage with its integrated backup and iSCSI approach uses tiers in exactly this manner. They use disk for sequential writes because disk is efficient for that purpose and service reads from the flash layer. They move data between tiers in 4K segments to provide a granular and fast block relocation capability. There is a good discussion of the design criteria and conclusions they reached here. XIO Storage is another provider of hybrid subsystems; XIO incorporates 200GB MLC SSDs from Smart Modular with 900GB SAS drives in a unique packaging approach called the Intelligent Storage Element. Using a map of subsystem activity called CADP (continuous adaptive data placement), I/O activity is monitored fulltime and hot blocks can be relocated in 5 second intervals to respond to changing requirements. This functionality corresponds well with the types of applications that are of current concern for the enterprise such as database acceleration, virtualization environments and VDI. An illustration of a capability similar to what is described here can be found in Oracles Smart Flash Cache, which was introduced in Oracle Database 11g Release 2. In that implementation, a Sun Storage F5100 Flash Array or Sun Flash Accelerator F20 is used to provide an intermediary storage location between the System Global Area in main memory and disk. This arrangement has the potential to significantly increase database performance for repeated short transactions with a large number of users, high read activity in the storage system, or when main memory is constrained. More information about this level of integration can be found here.

Will Alger

January 2012 10 | P a g e

Conclusions
There is a dizzying variety of possibility in the flash market as it relates to enterprise applications today, and we can be sure of only one thing- there will be more options, more claims and more confusion in the short term. It is best to remember what we originally set out to solve and not get sidetracked by what may appear to be the latest and greatest solution; there will lots of those, and of course not all of them will live up to the hype.

Some things that can be recommended in the evaluation of flash-based products; Ignore vendor claims of performance; do your own comparisons Look for real-world examples that closely correspond to your own needs Realize that support and commitment from your selected provider of storage are more important than other factors that may be temporarily desirable such as apparent low cost Scalability is important; if the upgrade path is buy another one then you need to look at other options Reliability is paramount; dont accept design compromises that can affect the integrity of your data Look for account references running applications similar to your own and gauge their experiences

Will Alger

January 2012 11 | P a g e