03.technical Overview For FlashSystem 900 & V9000

IBM FlashSystem 900 &
IBM FlashSystem V9000
Technical Overview
Firat Ozturk
IBM MEA Flashsystem Technical Sales Leader
Firat@ae.ibm.com
06.12.2016, Riyadh
Agenda
Introducing flash memory for the enterprise

Building an IBM FlashSystem, what makes it different?
Introducing IBM FlashCore Technology
Detailed look at the technology
FlashSystem V9000 architecture
FlashSystem V9000 Hybrid (New, as of 5/24/16)
IBM Corporation 2016 IBM & IBM Business Partner Use Only FlashSystem | 2
IBM Flashsystems
IBM has first introduced Flashsystem Family by 2012 after TMS (Texas Memory
Systems) acquisition. TMS is offering these Flash Based Storage systems since
1970s.
After the acquisition, IBM has invested 1B+ USD to Flashsystem Family. As of today,
IBM Flashsystem solutions are the only technology which has 35+ years of field
experience where others are around 3-7 years of field experience
Introducing Flash Memory For The Enterprise
A single flash chip
Flash Memory Cannot hold enough data
Physically small and Writes & erases slowly
design flexible Is not reliable enough
Inherently fast at reads
Non-volatile
Very low power
NAND Flash comes in many types
Inherently rugged TLC very close to read-only
MLC most cost-effective
SLC highest performing
NAND Flash density is increasing

More capacity per unit of space
Lower cost per capacity
More challenges for storage engineers
The Steps To Build An IBM FlashSystem
Array the flash chips

ECC, wear leveling and overprovisioning
Module-level RAID
System-level RAID
Reliability, availability, serviceability
Add features without compromising performance
Building A FlashSystem
A single flash chip But, a single flash chip
Fast at reads Cannot hold enough data

Non-volatile Writes & erases slowly
Very low power Is not reliable enough
Step 1 : Array The Flash Chips
Arrays of flash chips

Advantages:
Scalable Increases capacity
Fast at reads Increases I/Os per second and
bandwidth
Non-volatile
Very low power Problem:
Magnifies reliability problems
Write performance is not great
Who has it?

All enterprise flash products but, IBM
doesnt just use standard Flash.
Step 2 : Add ECC, Wear Leveling & Overprovisioning
Add ECC, Wear Leveling, O/P Advantages:

Improves wear life
Minimum level of flash Faster write performance
module data protection
Sufficient wear life Problem:
Fast writes Lose a flash chip and lose data
Scalable
Fast at reads
Non-volatile Who has it?
Very low power All enterprise flash products
but not all solutions are created equal:
varying degrees of ECC protection
different tools for wear leveling
percentages of overprovisioning
FlashCore: Advanced Flash Management
IBM Advanced Flash Management improves MLC flash endurance 9x
over standard implementations without sacrificing latency.
Backed by IBMs Flash Wear Guarantee!

Through close collaboration between IBM research, development and flash engineering, IBM created an
advanced flash characterization platform to test thousands of flash devices over a lifetime of wear. As a
result, IBM has made major strides in advanced flash management. Andy Walls, IBM Fellow
Standard MLC Flash Key accomplishments:

IBM Enhanced MLC Innovative ECC algorithm corrects very high bit error rates
Variable voltage and read level shifting maximizes flash endurance
Advanced Flash Management continually monitors the health of blocks
9.4X Endurance Improvement
and performs asymmetrical wear leveling and sub-chip tiering:
- Health binning: up to 57% improvement in endurance!
- Heat segregation: up to 45% reduction in write amplification!
Micron Technology provides special access to the chip
Step 3 : Add IBM Patented Variable Stripe RAID
Add Variable Stripe RAID Advantages:

Protects data from a chip failure
Maximum level of flash Dynamically re-stripes data at a sub-chip level
module data protection Preserves life, protection and performance
Maximum wear life
Fast writes IBM VSR protected flash modules:
Scalable Now we can build our system using far
Fast at reads more advanced technology than SSDs
Non-volatile
P Very low power
3 Who has it?
2 Only the IBM FlashSystem
1
Step 4 : Add Two-Dimensional (2D) Flash RAID
Add 2D RAID Protection
Advantages:
Maximum level of
system protection VSR protects from flash chip or sub-chip issues
Maximum level of flash System-level RAID protects against abrupt module
failure and controller failure
Maximum wear life
Fast writes
Scalable
Fast at reads
Non-volatile
Very low power Who has it?
Only the IBM FlashSystem
2D RAID
across
modules
VSR within modules
Step 5 : Add Reliability, Availability, Serviceability
Add Enterprise RAS
Advantages:
Enterprise reliability, VSR protects from flash chip or sub-chip issues
availability, serviceablility System-level RAID protects against abrupt
Maximum level of module failure and controller failure
system protection
Maximum level of flash
Maximum wear life
Fast writes
Scalable
Fast at reads Who has it?
Non-volatile
All enterprise storage products
Very low power should have it
Step 6 : Add Software Features (Where Needed)
Add Software Features
Advantages:
Advanced data Advanced software features when you need them
management Bare bones performance when you dont
Enterprise RAS Leverages industry leading data virtualization
Maximum level of
system protection
Maximum level of flash
Maximum wear life
Fast writes
Scalable Who has it?
Fast at reads
Only the IBM FlashSystem
Non-volatile
Very low power
Why IBM Flash ?
After acquisition, IBM analyzed all those field experience and excelled the product line by
IBM Research and Micron to meet every client expectations in every demanding workload
Unlike other Flash based solutions in the market today , IBM has developed a purpose
build Flash media which is called IBM MicroLatency Module. There is no software
running in the data-path of IBM Flashsystem to drive these modules
These modules are at the heart of IBM Flashsystem

Solutions. Theyre ;
o IBM Engineered (NOT commodity off-the shelf SSD drive)
o Massively Parallel Design
o Using hardware path designed for Flash, implemented using FPGAs
(NOT SAS/SCSI/FC)
o Distributed Ram
o Hardware based Data at Rest Encryption
Why IBM Flash ?
Flashs known chronic problems, Garbage Collection, Wear
For Garbage collection each IBM Microlatency Modules has dedicated FPGAs (other than IO
workers). System can do IO and collect garbage real time
Flash Cells die in time (wear-out) IBM can change the voltage level for semi-healthy cells,
which leads to longer life
o To overcome BER, ECC is required which
usually done by reads. IBM has changed this
with a novel code
o These technologies enabled IBM to have 9.4x

more endurance than SSD based systems
Why IBM Flash ?
Data Protection , 2D-Raid
Today , SSD based Flash systems, has to Software Based RAID rebuild the whole SSD drive if theres a
failure at one of the Flash chips in SSD drive. This creates a performance/latency penalty for those systems
IBM MicroLatency Modules , has 64 chips internally. A module can survive up to 17 chip failures and will
not give a warning till 10th chip has failed. After 11th chip failed a warning is raised and module continues
operation.
o Module has a Variable Stripe Raid (VSR)

protection along these 64 chips and every time
a chip fails , it dynamically re-stripes data
among other healthy chips
o IBM Flashsystem also maintains another RAID

protection across IBM MicroLatency Modules
Variable Stripe RAID
Patented Variable Stripe RAID allows RAID stripe sizes to vary.

If one die fails in a 16-chip stripe, only the failed die is bypassed, then data and parity is
restriped across the remaining fifteen chips (some chips on rear of circuit board).
Variable Stripe RAID reduces maintenance intervals caused by Flash failures
Flash
Flash Chips
Chips
Dea
FAI
dL
FlashSystem |
Why IBM Flash ?
Capacity Efficieny
IBM Flashsystem are using 3 Types of IBM MicroLatency Modules. These modules have RAW sizes of ; 1.8TB, 4.4TB and 8.8TB
and net usable sizes of; 1.2TB, 2.9TB and 5.7TB.
The 50% overprovisioning between RAW and NET is a sign of IBMs dedication to his Clients, these capacities are used for all those
Enterprise, RAS features mentioned before
IBM Flashsystems enable customers to use 100% of the net usable capacity
o You dont need to spare 10-15% of net

capacity for background tasks
o Also its very flexible and efficient, you can mix

and match different size modules in same
system while scaling
o Each shelf could be partially filled
Introducing IBM FlashCore
IBM uses tools and technologies to drive a robust solution
Industry
IndustryStandard
StandardInterface
InterfaceTechnology
Technology
Powered
Powered by
by
IBM FlashCore Technology
IBM FlashCore Technology
Hardware Accelerated I/O
IBM MicroLatency Module
Advanced Flash Management
IBM
IBMEnhanced
Enhanced
Micron
MicronTechnology
Technology
Scalable
Performance
V9000 Fully Integrated System Management Agile

Enduring
Economics
Agile Integration Integration
Thick and thin provisioned volumes

Snapshots, clones, backups, and replication (sync & asyc)
High availability configurations Active/active Hyperswap
Automated tiering
Data migration and preferred-read mirroring
Space efficient snaps
IBM Real-time Compression
Hypervisor integration
Quality of service
External virtualization
System View - Single Building Block
System View - Multiple Building Blocks
Ne
w
fo
r2
FlashSystem V9000 Tiered (Hybrid) Solution 01
6
Flash-optimized hybrid array for the tiered data center
Fast
Easy
Enterprise
Value
The Flash Sweet Spot
15% flash tier will handle ~90% of IO
12TB
12TB 29TB
29TB 57TB
57TB
Up to 2 expansions
80TB 13%
13% 27%
27% 42%
42% at initial GA (2Q16)
160TB 7%
7% 15%
15% 26%
26% Up to 20 expansions
240TB 5%
5% 11%
11% 19%
19% per FlashSystem
320TB 4%
4% 8%
8% 15%
15% V9000 (2H16)
percentage of flash tier
EasyTier performance:
Depends on access patterns (heat distribution) of the workload
Point of diminishing return for adding flash capacity
More Detailed Look At The Technology
FlashSystem Chip Terminology and Capacity Breakdown

Variable Stripe RAID
Two-dimensional (2D) Flash RAID Protection
FlashSystem Core Performance Concepts
Flash Module Architecture
System Architecture
Concurrent Code Load
FlashSystem V9000 Architecture
FlashSystem 900 Terminology & Capacity Breakdown
FlashSystem 900 57TB Usable Maximum 1 chip = 8 dies = 2 planes = 1K blocks = 512 pages = 16K bytes
Variable Stripe RAID (VSR)
4k 4k 4k 4k 4k 4k 4k 4k 4k 4k
Gateway
Interface
Flash
Controller
... ... ... ... ... ... ... ... ... ...
Patented Variable Stripe RAID protection allows RAID stripe sizes to vary
With traditional RAID 5, entire stripe has to be relocated
Variable Stripe RAID works hand-in-hand with ECC protection
As we layout the flash writes, we drop in parity across lanes
No cost in performance
Cost only 10% of RAW capacity
Bad blocks are simply ignored from future reuse
Variable Stripe RAID (VSR)
Form Factor SSD
Flash failure = Disk failure
Requires top-level RAID
Relatively frequent hot-swaps
No Parity Value of Variable Stripe RAID:
Enterprise Flash Drive or Memory Module

Flash failure = Degraded state
Less maintenance touches
within module while still preserving the life,
Performance impact on RAID set protection, and performance
Parity Hot-swap to resolve of the Day-1 experience
FlashSystem with Variable Stripe RAID

Preserves Flash life
Preserves performance
Re-parity data in microseconds
Parity
Two-Dimensional (2D) Flash RAID Protection
VSR provides 1st level of data
protection (module-level)
System-level RAID protects against
flash controller failure (abrupt module
failure)
RAID-5 across modules, hot
swappable from front of chassis
Two independent crossbar (XBAR)
controllers
Interface controllers stripe across
XBARs
Granular capacities configurable with
4, 6, 8, 10 or 12 flash modules
FlashSystem Core Performance Concepts
Interface Modules
FC FCoE IB FC FCoE IB
Management Modules
CPUs (18)
RAID Modules
Flash Modules (12)
FlashSystem provides a hardware-only data path

Custom FPGA-based data movement decreases latency vs. software
Lower latency on standard SAN interfaces vs. competitors
Distributed out-of-data-path CPU processing
Flash Module Architecture
FPGA FPGA
FPGA
FPGA FPGA
Primary Board Secondary Board

(2.9TB) (2.9TB)
FPGA
FPGA
Gateway Interface FPGA Flash Controller - FPGA
I/O interface and Direct Memory Access path 2 or 4 per Module
Data path, Hardware I/O logic
Look up Tables and Write Buffer
Each controls 16 flash Chips
Control PPC and DRAM
Out of Data path operations Flash Chips
Garbage collection, Error Handling, System Health 16 per Flash Controller
Wear Leveling, Statistics, etc. 32 or 64 per Module
Data XOR parity
Scalability and Parallelism
FPGA FPGA
FPGA FPGA
Up to 4 Controllers per Module
Each can do 40 4K DMA operations in parallel
Up to 44 Active Controllers per FlashSystem 900

1,760 4K DMA (direct memory access) operations in parallel
The FlashSystem 900 can also issue 8 concurrent

operations per flash chip.
With up to 64 chips per flash module and 11 active flash

modules per system, the FlashSystem can service up to
5,632 simultaneous operations in flash.
IBM FlashSystem 900 Components
IBM MicroLatency Modules (12)
Battery Packs (2)
RAID Controllers (2)
Interface Modules (4)
Fan Packs (4) Canisters (2)
Management Modules (2)
Power Supplies (2)
IBM FlashSystem 900 Back
Interface card 1 Interface card 2 Interface card 3 Interface card 4
Power supply module 1
Power supply module 2
Fan module 1 Fan module 2 Fan module 3 Fan module 4

RAID module 1 RAID module 2
IBM FlashSystem 900 Canisters
Entire canister is a field

replaceable unit
Interface Modules
Four interface cards per system
FC, iSCSI, & FCoE InfiniBand
Each with:
Four ports 8Gb FC Two ports 40Gb IB
Two ports 16Gb FC
Four ports 10Gb FCoE
Four ports 10Gb iSCSI
Concurrent Code Load
The data path is fully redundant and has no single points of
failure 1
1 2
2
Management Management
The data path is entirely independent of the control path Module
(FPGA & CPU)
Module
(FPGA & CPU)
Disruption in the control path does not adversely affect the data
path
Canister components connect to the other canister components
In particular, each Interface Card is attached to each Crossbar
Each Management Module can directly monitor each component in
both canisters
During CCL components are upgraded in a smaller granularity

than the canister allows
Concurrent Maintenance, uses similar procedures used in CCL
IBM FlashSystem V9000 Architecture
Control Enclosure (AC3)
Overview
Same controller hardware
6U bezel hides two 2U controllers and one 2U flash enclosure
Scale up and scale out
Latency (4K) 200s Scale-up

Maximum Performance IOPS (4K) 2,520,000 (add capacity)
Scaled Out
(100% Read, 4 Building
Bandwidth (128K) 30 GB/s
Blocks)*
Latency (4K) 200s

Maximum Performance IOPS (4K) 630,000
Scalable Build Block
(100% Read, Cache Miss) Bandwidth (128K) 9.5 GB/s
Scale-out
Latency (4K) 200s (add capacity
Maximum Performance IOPS (4K) 630,000 and performanc
Fixed Build Block
(100% Read, Cache Miss) Bandwidth (128K) 9.5 GB/s
2016 IBM Corporation IBM | 39

Scalable
Performance
FlashSystem V9000 Options For Deployment

Scalable Performance
Application Accelerator
Small Data Center
Great for large data sets with
Great for database and VDI big I/O requirements and
workloads needing storage services
Accelerate, migrate, tier, Scale up for more all flash
clone, snapshot, replicate, Public or Private Cloud capacity
compress existing storage
Great for tier 1 disk Up to 630K IOPS, 200s
Up to 630K IOPS, 200s
replacement Up to 285TB usable, 1.4PB
Up to 57TB usable, 285TB
Scale up and out for
Effective
effective
more processing, more
capacity and more I/O Virtualized Data Center
Mixed Workload Accelerator Up to 2.5M IOPS, Great for data centers with
200s heterogeneous storage
Great for multiple mixed Up to 456TB usable, Extends core feature set to
workloads that drive huge I/O 2.2PB Effective other storage arrays
Scale out for more all flash Up to 2.5M IOPS, 200s
capacity, IOPS and bandwidth Up to 3 Exabytes virtualized
Up to 2.5M IOPS, 200s
Up to 228TB usable, 1.1PB
Effective
Quality of Service (QoS)
Help alleviate performance

contention
Give your application the

performance it needs
Ability to cap IOPS performance

on a per volume basis

Fully integrated system management
Improve workforce productivity
Simplify management
Single name space

Flexible volumes
Generic fully allocated

Thin-Provision space consumed on-demand
Mirror two copies of volume
Migrate between pools or change
volume type
Preferred read mirror between flash
and legacy storage (all reads from flash)
Thin mirror a mirror with copies being thin
Compressed volumes utilizing Real-time Compression

Non-disruptive data migrations
Non-disruptive data migration

Maintain concurrent I/O access
Externally virtualize the existing
storage system
Move data off older storage onto
flash storage

FlashSystem + VMWare: an ideal fit
Driving efficiency, performance, flexibility

& scale in virtualized environments
Area Integration Value

Redefine tier 1 performance
Managemen vCenter plug- Storage visibility
for workloads with diverse I/O t in Self-service provisioning with controlled
delegation
patterns
Business SRA for Site Automated Storage and Host
Continuity Recovery Failover
Optimize your data tier with Manager Failover testing
Failback
thin provisioning, snapshots, Performance VAAI Hardware accelerated VM
and real-time compression copy/migration
Hardware accelerated VM initiation
Accelerate VMFS (no SCSI reservation)

FlashSystem + VMWare: vSphere 6.0 vVols
No shared storage
1-to-1 mapping of VMs drives to FlashSystem V9000 volumes
No IO contention or VMFS overhead
Automation
vVols are created/deleted automatically when required
Hardware acceleration
Off-loading snapshot & cloning functionality to the storage level
Granularity
More granular visibility, isolation and adjustment of VM storage
Policy Based Storage Management

Shorten provision cycle with simpler automation
Improve SLAs with application aware storage provisioning, monitoring, and optimization
= speeds things up

Flash for less than the cost of Tier 1 disk
Real-time Compression, for active and inactive data

Traditional
Virtual
Database JPEG/ZIP Geoseismic CAD/CAM
Storage Office Files Servers
Compression 80% 30% 5% 50% 72% 70%

Rate
Real-time Compression Efficient Average of

Half the Data Storage 65% Compression

Storage virtualization
Preserve your existing Manage

Tier
investment in storage Protect
Compress with Real-time Compression
Flash HDDs Flash HDDs
IBM Easy Tier flash storage management

Optimized performance at lower overall cost
Business continuity
Full set of disaster recovery tools:

FlashCopy
FlashCopy
Snapshot Instant space-efficient, Metro Mirror Synchronous Options
Options
Updates are committed to both sites Full or thick
thin provisioned copy, capture state of before acknowledged to server
Full or thick
Thin nocopy
volume before a change Thin nocopy
Supported up to 300 km, adds Incremental
additional latency to hosts Incremental
Cascading
Clone Creates a complete one-time Cascading
Multi-target
replica of source, testing data or Global Mirror Asynchronous Multi-target
Consistency groups
Consistency groups
attaching to other servers Server update acknowledged in
Reverse
Reverse
parallel to update to remote site
Backup Complete point in time Supported up to 80ms of round-trip
latency, distance dependent upon
replica that can be incrementally network quality
refreshed

Introducing IBM HyperSwap (Active/Active Data Center)
Provides highly available, concurrent

access to a single copy of data from
data centers up to 300 km (186
miles) apart
Enables non-disruptive storage and
virtual machine mobility between
data centers
Works with standard multipathing
drivers
Groups multiple volumes together for
high availability
Requires the Remote Mirroring
2016 IBMlicense
Corporation IBM | 50
Oracle workload environments
Customer Real Time

E-commerce Analysis Fraud Detection
Scalable Transactional Database Analytics Data Warehouse Operational Data Warehouse
Transaction Processing Reporting and Analytics Operational Analytics

Random reads & sequential data Random and sequential reads &
Random reads & updates
loads data loads + continuous ingest
Many transactions with narrow data Analytics split into many parts and
Analytics with broad data scope,
scope accessing the same narrow scope operations,
parallelized across data partitions
database all running in parallel
Requires low latency and high IOPS Requires low latency and high Requires low latency and high IOPS
bandwidth
FlashSystem |
Datacenters Response to Bridge Disk Performance Gap
Wasteful,
HDD Expensive &
Most Costly & Add More
Performance Ineffective with
Volatile Memory
Enhancement Storage Latency
Issues
Typical
Performance
Mitigation
Tactics
Expensive &
Ineffective for
Tune & Time Consuming,
Add CPUs Modify Very Expensive &
Storage
Application Risky
Performance Issues
These solutions do not address the real cause of

performance bottlenecks: LATENCY
FlashSystem |
IBM Flashsystem for Oracle
Some real life data

FlashSystem V9000 Software Licensing
Per enclosure
Base Software Internal
(5639-RB7)
- Easy Tier Incl
- FlashCopy Incl
- Remote Mirror Incl
- Compression Incl
Per enclosure
VSC Internal (5608-ACL)
VSC External Per TB

(5608-AE1)
When purchased separately:
- External Virtualization Per TB (5641-VC7 0663)
- Remote Mirror Per TB (5641-VC7 0679)
- FlashCopy Per TB (5641-VC7 0671)
- Easy Tier Incl

Compression External Per TB (5641-CP7 0708)
VSC includes: External Virtualization, Easy Tier, FlashCopy, and Remote Mirror + TPC, Storage Analytics Engine, and FlashCopy Manager

03.technical Overview For FlashSystem 900 & V9000

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

03.technical Overview For FlashSystem 900 & V9000

Загружено:

Авторское право:

Доступные форматы

IBM FlashSystem 900 &

IBM FlashSystem V9000

Introducing flash memory for the enterprise

NAND Flash density is increasing

Array the flash chips

A single flash chip But, a single flash chip

Fast at reads Cannot hold enough data

Arrays of flash chips

Who has it?

Add ECC, Wear Leveling, O/P Advantages:

Backed by IBMs Flash Wear Guarantee!

Standard MLC Flash Key accomplishments:

Add Variable Stripe RAID Advantages:

These modules are at the heart of IBM Flashsystem

o These technologies enabled IBM to have 9.4x

o Module has a Variable Stripe Raid (VSR)

o IBM Flashsystem also maintains another RAID

Patented Variable Stripe RAID allows RAID stripe sizes to vary.

o You dont need to spare 10-15% of net

o Also its very flexible and efficient, you can mix

IBM MicroLatency Module

Advanced Flash Management

V9000 Fully Integrated System Management Agile

Agile Integration Integration

Thick and thin provisioned volumes

Flash-optimized hybrid array for the tiered data center

percentage of flash tier

FlashSystem Chip Terminology and Capacity Breakdown

Enterprise Flash Drive or Memory Module

FlashSystem with Variable Stripe RAID

Flash Modules (12)

FlashSystem provides a hardware-only data path

Primary Board Secondary Board

Up to 44 Active Controllers per FlashSystem 900

The FlashSystem 900 can also issue 8 concurrent

With up to 64 chips per flash module and 11 active flash

RAID Controllers (2)

Interface Modules (4)

Fan Packs (4) Canisters (2)

Management Modules (2)

Power Supplies (2)

Power supply module 1

Power supply module 2

Fan module 1 Fan module 2 Fan module 3 Fan module 4

Entire canister is a field

FC, iSCSI, & FCoE InfiniBand

During CCL components are upgraded in a smaller granularity

Latency (4K) 200s Scale-up

Latency (4K) 200s

2016 IBM Corporation IBM | 39

FlashSystem V9000 Options For Deployment

Help alleviate performance

Give your application the

Ability to cap IOPS performance

2016 IBM Corporation IBM | 41

Improve workforce productivity

Single name space

2016 IBM Corporation IBM | 42

Generic fully allocated

2016 IBM Corporation IBM | 43

Non-disruptive data migration

2016 IBM Corporation IBM | 44

Driving efficiency, performance, flexibility