CASE STUDY: ELASTIC CLOUD

STORAGE (ECS) & VMWARE

VCLOUD AIR - ARCHITECTING &
DEPLOYING AN EXABYTE + CLOUD

CHRISTIAN ARANGIO AND BRADEN GIBSON

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ROADMAP INFORMATION DISCLAIMER
EMC makes no representation and undertakes no obligations with
regard to product planning information, anticipated product
characteristics, performance specifications, or anticipated release
dates (collectively, Roadmap Information).
Roadmap Information is provided by EMC as an accommodation to the
recipient solely for purposes of discussion and without intending to be
bound thereby.
Roadmap information is EMC Restricted Confidential and is provided
under the terms, conditions and restrictions defined in the EMC Non-
Disclosure Agreement in place with your organization.

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
TRADITIONAL STORAGE REALITIES

BLOCK FILE FILE BLOCK FILE OBJECT HDFS

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
DATA GROWTH IS BREAKING
TRADITIONAL STORAGE
SCALE-OUT FILE & OBJECT STORAGE CONTINUES TO GROW

Overly complex
Multiple data protection schemes,
protocols, management tools

Cant economically scale

Inefficient, high overhead,
especially at geo-scale

Not cloud-ready
Not architecturally suited and no
self-service

Source: IDC EMC Digital Universe Study 2014 #EMCECS

Copyright 2015 EMC Corporation. All rights reserved.
MODERN APPS ARE BREAKING
TRADITIONAL STORAGE
NOT DESIGNED FOR CLOUD AND BIG DATA APPLICATIONS

Architecture is too complex

Locking, replication, High Availability, geo-
distribution is complex

Not Web or firewall friendly

Distributed (WAN) access is complex

Storage silos impede development

Different hardware for every data type and
access protocol

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ITS ONLY GOING TO GET WORSE

NEW PLATFORMS & END USER DEMAND MASSIVE DATA

APPROACHES FOR CLOUD GROWTH

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
HOW DO WE SOLVE FOR THIS?

COST CHOICE CLOUD

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
SOFTWARE-DEFINED STORAGE
TURN COMMODITY SERVERS INTO A STORAGE SYSTEM

Software-Defined
Storage

Commodity
Platforms
#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
WHATS A DIME WORTH?
$0.10/GB SAVINGS = $100,000,000 / EXABYTE

1 Terabyte

1 Petabyte

1 Exabyte

$1 $10 $100 $1,000 $10,000 $100,000 $1,000,000 $10,000,000 $100,000,000

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
COMMODITY HARDWARE VALUE
PROPOSITION
Utilize standardized, open
technologies and mass market
components
Individual components provide
lower performance, reliability, etc.
At sufficient scale, with the right
software, the component pool
provides superior characteristics

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
OBJECT STORAGE IS DESIGNED FOR
CLOUD INFRASTRUCTURES
1. Shared Utility single storage system & access point across
many global apps
2. Frictionless Scale No need to manage client and server
endpoints (e.g., LUNs, mounts, etc.) as apps grow or resources
change
3. REST (HTTP-based) APIs Internet, firewall, and mobile device
friendly
app app app

https://accesspoint.yourcompany.com

EMC ECS Cloud Storage #EMCECS

Copyright 2015 EMC Corporation. All rights reserved.
BUILD A MODERN HYPERSCALE
STORAGE INFRASTRUCTURE USING
COMMODITY PLATFORMS

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
DELIVER STORAGE ON COMMODITY
PLATFORMS

ECS
Software OBJECT HDFS
STORAGE STORAGE

Commodity
Platforms
#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ECS SOFTWARE ARCHITECTURE
CLOUD-SCALE STORAGE SERVICES ON COMMODITY

ECS
Software OBJECT HDFS
STORAGE STORAGE

Geo-Replicated Data Protection

Active-Active read/write support with strong consistency
No single point of failure
Performance and efficiency for small and large objects

SITE 1 SITE 2 SITE 3

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ECS: WEB SCALE DATA STORAGE
Modern Application Global Content Geo-scale big Data
platform Repository Analytics

https://accesspoint.yourcompany.com

Turnkey Public, Geo-scale efficiency

Private Cloud Storage & access

Lower $/GB than Memphis Broadest application

public IaaS L.A. U.K. support

#EMCECS
Complete Cloud Storage Platform
Copyright 2015 EMC Corporation. All rights reserved.
ECS Appliance Highlights

Uses COTS Components

Economies of scale

Density Optimized
Up to 72TB Raw / Rack Unit
Saves Power/GB, Real Estate costs, etc.

Labor Optimized
Manage the cluster, not the devices
Maximize Serviceability

Protection Efficiency
Geo-efficient storage

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ECS Appliance Hardware Configurations

Rack Based Deployments

Software Clustered within and across
racks
Up to 3PB raw capacity per rack

Components
Services Nodes (4 or 8 per rack)
Disk Enclosures (4 or 8 per rack)
Disks (up to 480 per rack)
Switches (2x10GBE + 1x1GBE)

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
THINKING IN CLOUD
The Traditional Way
Identifiable users/apps for specific
infrastructure
Monitor and manage discrete
systems and components
Tune or customize infrastructure
according to the needs of the app
Offline maintenance and windows for
service / upgrade / migration
Boundary security and implicit trust
Copyright 2015 EMC Corporation. All rights reserved.
The Cloud Way
Common, shared infrastructure
Focus on the overall service level
No maintenance windows no such
thing as a planned outage
Need online operations for full
infrastructure lifecycle (upgrade,
migration, break/fix)
Secure all interactions to address
both external and internal use cases
and threats
#EMCECS
EMC CLOUD SERVICES - STORAGE

Object Storage as a Service - AWS S3 API (subset)

Target use cases: Archive, Cloud Tier Storage, Media, BLOB
Internally monitored 24x7
Geographic Zones (US 2014 & EMEA - 2015)
Alternate Site Data Replication within Zone
ViPR replication explained (technical overview) here
3-9s Availability / 11-9s Durability
Security Compliant SSAE-16
#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
GEOGRAPHIC ZONES & REPLICATION

Data is replicated across multiple locations within

each Geographic Zone
Ex: US Data in Nevada, Virginia, California, and New Jersey
Presented as one large storage cloud with one public name
Multiple public names can be mapped to the same cloud
Each Geographic Zone is a separate URL
Within the Geographic Zone
Requests are assigned to the nearest available center
Data is replicated across the locations
Each location has local protection of the data
#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ECR OVERVIEW
CLOUD STORAGE API ROUTER

Firewall
Basic Web API Filtering
SSL Termination
Load Balancing
Request Validation
Bucket/Object Hashing
Identify Target Cloud(s)
Hash Read / Write Lists
Generate New Request(s)
Connect Client to Target Cloud

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
SHARDING
Send OP to Write target
POST
PUT (HEADS 404)
C

HEAD

RUD

HEAD
GET
PUT
DELETE
HEAD to Read List
Send OP to 200 Responder #EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
MIGRATING

GET
HEAD
Source

GET
Cache Object

PUT

DELETE

Destination

Creates on Destination
Updates in Place (HEAD then update)
Overwrites might be possible to direct to Destination
#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
 TRANSLATING

Atmos API

S3 API

Que?!?
S3 API

Mais Oui!
Parlez vous Atmos?
#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
OPERATING Read Only for Troubled Clouds
Creates
Balance Cloud Capacities

Queue DELETEs during high load Obscure Scheduled Maintenance

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
 GEOGRAPHIC ZONE ARCHITECTURE
OVERVIEW
DNS - LB DNS - LB DNS - LB DNS - LB

Cluster (site) Cluster (site) Cluster (site)

ECR ECR ECR ECR ECR ECR ECR ECR ECR
ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ECS Rack ECS Rack ECS Rack ECS Rack ECS Rack ECS Rack ECS Rack ECS Rack ECS Rack

Replication #EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
 GEOGRAPHIC ZONE ARCHITECTURE
OVERVIEW
TRANSACTION VIEW
ViPR Node ViPR Node ViPR Node
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node

Determine Closest Site 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks
Select an ECR in Site
ViPR Node ViPR Node ViPR Node
Return ECR IP Address
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

DNS ViPR Node ViPR Node ViPR Node

DNS - LB
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

ViPR Node ViPR Node ViPR Node

Client

60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

HTTP(S) New Headers
ViPR Node ViPR Node ViPR Node
ECR* Original Payload 60 x 6TB disks 60 x 6TB disks 60 x 6TB disks

Response ViPR Node ViPR Node ViPR Node

60 x 6TB disks 60 x 6TB disks 60 x 6TB disks
Terminate SSL
ViPR Node ViPR Node ViPR Node
Verify Signature
60 x 6TB disks 60 x 6TB disks 60 x 6TB disks
Select Shard


Select ViPR Node
Regenerate Headers
ECS Rack ECS Rack ECS Rac
Connect Payload
Connect Response * Elastic Cloud Router #EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ECS AT EMC WORLD
ECS is Everywhere at EMC World!
On The Show Floor
ECS at CTD/ETD Booth 813 and kiosks #8 and #9

Breakout Sessions
EMC ECS: ECS and Unstructured Content Strategies for Web
Scale IT
EMC ECS: ECS and VMware vCloud Air - Architecting and
Deploying an Exabyte+ Cloud
EMC ECS: ECS Architecture Deep Dive
EMC ECS: ECS Storage for 3rd Platform and What's New in
2015

HOLs (Hands-on Labs)

EMC ECS: Empowering Storage Admins for Cloud Storage

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.