White Paper

Advantages of Fibre Channel Realized With Low-Cost SATA Disk Drives Silicon Storage Router Family
At A glAnce This white paper presents the advantages of system-level approach employed by Emulexs Fibre Channel (FC) to Serial ATA (SATA) Silicon Storage Router in enabling the deployment of lower-cost SATA storage systems in the FC storage infrastructure. Products SR-1216 Silicon Storage Router APPlIcAtIons JBODs, SBODs & Tiered Storage Architectures

Introduction
Today, the vast majority of JBODs shipped into enterprise storage applications are based on Fibre Channel technology. They are built on an FC-AL infrastructure, use Fibre Channel connectivity to RAID controllers and rely on high-priced, dual-ported Fibre Channel disk drives. They provide high performance, reliability and availability, but at a high Total Cost of Ownership. Emulexs innovative Silicon Storage Router line of controllers breaks through the cost barrier that prohibits the use of expensive FC disk drives in all but the most critical enterprise applications. The Silicon Storage Router line of controllers accomplishes this breakthrough by enabling the use of lower-cost SATA disk drives in integrated storage solutions that leverage Fibre Channel as the connectivity to the RAID controller and between JBODs. At the same time, the Silicon Storage Routers increase storage system availability, functionality and performance with little modification to existing storage system firmware. The result is costeffective, versatile, scalable, transparent solutions that justify moving high performance RAID, MAID, NAS and disk-based backup systems into Small-to-Medium Business markets.

Architectural Highlights
The key architectural accomplishments of the Silicon Storage Router family of IC controllers:
c

Implement a unique methodology that allows SATA drives to emulate FC drives Deliver High Availability enterprise storage solutions Leverage existing data integrity measures

c c

Provide solutions previously available only from loop switches Increase the overall performance level of the FC loop

The following figure illustrates how the Silicon Storage Router controllers allow FC and SATA drives to be intermixed in a JBOD behind a common RAID controller with FC used to interconnect the RAID controller and the JBOD.

5 minutes per year or less than 30 seconds per month. This requirement can be satisfied only by a design that is fault-tolerant and self-healing. Hardware failures which are inevitable in any technology must be automatically detected and resolved without operator intervention. High Availability applications require that two complete but separate data paths exist from the end users application to the disk drives. Todays all-Fibre Channel solutions accomplish this by using dual-ported drives and a pair of redundant FC loops, both of which have access to all the devices on the FC-ALs. The primary challenge to using SATA HDDs in FC applications is that SATA HDDs are single-ported. A creative approach is needed to establish dual paths to all single-ported as well as dual-ported devices. Furthermore, all SATA HDDs that is, the emulated FC HDDs must be configured and accessible on both FC-ALs. In a High Availability SATA JBOD that uses Silicon Storage Routers, fault tolerance is accomplished with two highlevel features. First, dual Silicon Storage Router cards are installed in the JBOD. The two Silicon Storage Router cards operate in a load-balanced arrangement, with each card taking responsibility for managing half the drives in the SATA JBOD. I/Os are routed between the Silicon Storage Router cards so that each I/O arrives at the card that has direct management access to the associated SATA HDD. If one Silicon Storage Router card fails, the surviving card immediately and automatically assumes full responsibility for managing all the HDDs in the JBOD. This mechanism, which is transparent to the RAID controller, is an important part of the overall system solution that emulates the topology of a native FC JBOD and it is effectively selfhealing to keep the JBOD continuously online and available. The second high-level feature takes the form of an interposer card that is mounted adjacent to each SATA HDD within the replaceable drive canister. The interposer card implements the second, redundant, physical port to each single-ported SATA HDD and yields the dual paths to the redundant I/O controllers that are required in an HA

The innovative Silicon Storage Router architecture creates a new paradigm for JBODs; one that evolves from a nonintelligent box to an implementation that is a system in its own right. This system-level approach to the architecture of the Silicon Storage Router line of ICs and accompanying firmware yields an extremely powerful, proven solution that addresses the needs of storage OEMs. The main advantages of this system-level approach are:
c c c c

High Availability and Fault Tolerance Data Integrity Diagnostics Performance and Scalability

These advantages are detailed in the following sections.

HA and Fault tolerance

For a JBOD or RAID system to be enterprise-capable, it must be able to deliver virtually continuous up-time or availability. A common measure of availability is the percentage of time that data is available to end users. Enterprise applications typically require the product to be online and available 99.999% of the time, which equates to a downtime of about 

White paper
environment. The following figure shows a SATA HDD with an interposer card installed. expanded sector size, for example 520 bytes per sector. When expanded sectors of this size are used, the extra 8 bytes per sector can contain protection information for the 512 bytes of user data. The protection information is typically vendor-specific, but in most cases it is designed to ensure that the data is not corrupted and that the wrong block of data is not inadvertently accessed. In a native FC JBOD solution the application may require that the FC HDDs be formatted with a media sector size of 520 bytes. A problem arises when trying to employ this algorithm in a SATA JBOD because currently available, high volume SATA HDDs can be formatted only with a media sector size of 512 bytes. The Silicon Storage Router line of products solves this problem by differentiating user sectors from media sectors. Media sectors can be set to 512 bytes at the same time that user sectors can be set to 520 bytes, with the additional 8 bytes containing protection information that ensures data integrity. The Silicon Storage Router products translate between the 520-byte user sectors and the 512-byte media sectors in a high-performance and space-efficient manor. A 250 Gbyte disk drive using 520-byte user sectors delivers 246 Gbytes of useable space. This is identical to the efficiency provided by a 250 Gbyte FC drive formatted with 520-byte media sectors. The result is that no data integrity is sacrificed in enterprise storage applications that use low-cost SATA drive technology with Silicon Storage Router devices.

By using dual-ported SATA HDDs and two Silicon Storage Router cards the topology attains full path redundancy within the SATA JBOD, from the boxs external ports all the way to the HDDs. This concept is illustrated in the following figure.

data Integrity
To consider using a system in the most demanding enterprise storage applications, there can be no question about the safety and integrity of the data on that system. The user must be confident that data corruption is virtually impossible. A number of mechanisms are deployed in FC storage systems around the world to ensure data integrity. The Silicon Storage Router line of products enables these mechanisms to be used in SATA JBODs. The most common mechanism to ensure data integrity uses expanded sector sizes on the drive medium. In block-based storage, user data is stored in 512-byte blocks. Disk drives can be formatted to fit exactly one block of user data into each sector or they can be formatted to an

diagnostics
A significant weakness of previous PBC-based FC JBODs is the inability of the PBC infrastructure to diagnose and isolate problem components, including HDDs. Native FC JBODs solve this problem by using FC-AL switch devices that effectively instrument and isolate the behavior of any installed FC HDD. The Silicon Storage Router products use a similar approach, except for SATA HDDs. The storage OEM has the best of both worlds with this solution: Superior 

diagnostics and fault isolation while using significantly less expensive SATA HDDs. The Silicon Storage Routers continuously monitor and gather operational statistics on a per-HDD and per-link basis. At any moment in time several vital system metrics, such as link error rate, link reset count and I/O retry count are precisely known. These and many other metrics are accumulated by the Silicon Storage Routers and made available to the Enclosure Services subsystem within the SATA JBOD. The SATA JBOD, which typically employs a mechanism identical to its FC JBOD counterpart, in turn makes them available to the RAID controller and other higher level management functions. The key to the success of the Silicon Storage Router system is that it makes very detailed operational indicators available and it isolates faulty hardware.

The Silicon Storage Router also offers the ability to place up to 2,000 HDDs on each FC-AL. This is an important feature that FC loop switches do not yet offer.

sr-116 silicon storage router

The SR-1216 Silicon Storage Router is a sophisticated, highly-integrated, FC-to-SATA controller IC and the first member of Emulexs line of Silicon Storage Router products. In production and available to OEMs today, the SR-1216 delivers all the benefits described above and enables extremely cost-effective SATA storage solutions to work seamlessly in any enterprise storage environment. The SR-1216 provides the following high-level features on a single IC.
c c c

Two 2Gb/s FC ports Sixteen 1.5Gb/s SATA ports Complete SATA Initiator per port Intelligent management engine per SATA HDD Eighteen embedded SerDes Embedded ARM-9 microprocessor Integrated memories Data Path Processor Data and Instruction FC FIFOs 35 mm body size, 788 EPBGA-T package

Performance and scalability

It has been well documented that a FC loop begins to suffer in performance particularly small-block performance when the node count per loop passes a certain threshold. The solution to this problem is to isolate a large number of HDDs behind a much smaller number of node delays on the FC-AL. FC loop switches do this for a native FC JBOD. A Silicon Storage Router accomplishes the same thing for SATA HDDs. In a typical SATA JBOD configuration each SATA HDD is represented as a virtual FC device that consumes one FC-AL address (ALPA) per drive. This topology matches that of native FC JBODs and provides for direct leverage of todays existing FC RAID controllers. What is important to note, however, is that a Silicon Storage Router will consume up to 16 ALPAs per loop (one ALPA per SATA HDD), but the node delay of that SATA JBOD on the FC loop is reduced to just one a realized 16:1 improvement in terms of loop node delays. The practical benefit from this is the ability to build configurations of larger numbers of HDDs on a single loop. 

c c c c c c c c

The following diagram shows the high-level architecture of the SR-1216 Silicon Storage Router.

White paper
The SR-1216 Silicon Storage Router has integrated into a single device the functions performed by several separate, and expensive, discrete components. These functions include: c Dual-port FC controller
c c c c c

Dual, quad or octal SATA controllers Microprocessor Bus controller Data path memory Processor instruction and data memories

The integration of the SR-1216 is so complete that the only external IC needed for operation is a Flash memory chip. No processor or data path memories are required external to the SR-1216. A Silicon Storage Router Card, which is illustrated in the following diagram, is constructed of the following major components: c SR-1216 Silicon Storage Router IC
c c c c c

Flash FC hub IC (for example, 4-port PBC) SES subsystem (for example, SES processor, I/O) Mid-plane connector FC connectors

The performance improvements are a result of a significant architectural advantage the SR-1216 has over typical discrete designs, namely the lack of a store-and-forward mechanism. The SR-1216 employs a highly efficient flowthrough architecture that minimizes latency and maximizes throughput. Examples of both types of data flow mechanisms are shown in these two diagrams. The diagram above illustrates the traditional discrete store-and-forward mechanism. The diagram below shows the SR-1216s improved flow-through solution to moving data.

The SR-1216 provides many benefits over a discrete, nonintegrated approach. The primary benefits include: c Lower component cost
c c c c c

Lower component count Smaller form factor Significantly less power Improved feature set Increased performance
c c c c

Up to 35,000 IOPS (I/Os per second) 180-195MB/s per channel 360-390MB/s per JBOD Up to 5X performance boost over discrete implementations 

White paper
As can be seen in the discrete design that does not use the SR-1216, every frame received from the FC link first must be completely received and written into DMA (1). Then the microprocessor must analyze the frame and determine its destination (2). Next, the microprocessor must program the SATA protocol chips to move the frame (3), and finally, the data is retrieved from memory and transmitted onto the SATA links and to the HDDs (4). the SR-1216 routes the frame through the chip and out the appropriate SATA port. As can be seen, a data frame can still be inbound into the SR-1216 as it already is being written into the SATA HDD. This SR-1216 flow-through solution eliminates the long latencies inherent in the store-andforward approach. This dramatic difference in data flows between the SR-1216 and a traditional discrete design explains the significant performance increases demonstrated in real-life end-user operations. A discrete design is typically limited to less than 8K IOPS, while the SR-1216 can easily top 35K IOPS, a more than 4X improvement.

summary
The SR-1216 Silicon Storage Router, in production and available today, is a foundation technology solution that enables a new paradigm of enterprise storage systems. With this solution, IT managers have more cost-effective storage options available to them without sacrificing critical factors such as availability and data integrity. Thanks to industry-leading integration levels, the SR-1216 demonstrates impressive performance levels using lowcost SATA disk drives and delivers a complete, system-level solution to storage OEMs worldwide.

In comparison, the data flow for the same scenario using the SR-1216 is much more streamlined and efficient. As a FC frame is received, hardware within the SR-1216 recognizes that it is a data frame destined for one of its SATA HDDs, so

This document refers to various companies and products by their trade names. In most, if not all cases, their respective companies claim these designations as trademarks or registered trademarks. This information is provided for reference only. Although this information is believed to be accurate and reliable at the time of publication, Emulex assumes no responsibility for errors or omissions. Emulex reserves the right to make changes or corrections without notice. This report is the property of Emulex Corporation and may not be duplicated without permission from the Company.

www.emulex.com

corporate HQ: 3333 Susan Street, Costa Mesa, CA 92626 (714) 662-5600 Wokingham, u.K. (44) 118-977-2929 | Paris, France (33) 41-91 19-90 Beijing, china (86 10) 6849-9547 07-218 10/06