Network Scheduling Services for Emerging Applications

Nasir Ghani Advanced Cyberinfrastructure Lab ECE Dept, University of New Mexico Albuquerque, NM, USA http://ece.unm.edu/~nghani

Frontiers in IT (FIT) 2011 Islamabad, Pakistan December 19th, 2011

Advanced Cyber-Infrastructure Laboratory

Outline
Background Key Developments Scheduled Services Design Ongoing Efforts Conclusions

Advanced Cyber-Infrastructure Laboratory

Introduction
Advances in high-speed networking technologies:
Packet/frame switching @ layers 2, 3:
IP/MPLS, Carrier Ethernet

Circuit-switching @ layers 1, 1.5:

Wavelength division multiplexing (WDM), next-gen SONET

Traffic engineering (TE) control standards:

Generalized MPLS (GMPLS), path computation, etc

Focus shifts to applications

Leverage infrastructures to support client needs Surge in e-science & commercial demands Diverse service needs: scalability, flexibility, velocity

Advanced Cyber-Infrastructure Laboratory

Backbone Networks
Streamlined Architectures
IP, Ethernet core

IP/MPLS or Ethernet (Layers 2-3)

Multi-service IP QoS, advanced TE protocols

Carrier Ethernet solutions (PBB-TE, T-MPLS) Line rates up to 100 Gbps and beyond
OC-n/STS-nV, OTN

Next-Gen SONET/SDH, MSPP

NGS ring/mesh
10, 100 GbE, ITU-T G.709

Robust non-TDM mappings (GFP)

Efficient capacity matching (VCAT) Dynamic bandwidth control (LCAS) Grooming, Ethernet-over-SONET

OC-n/STS-nV, OTN

Optical DWDM Transport/Switching

Large tributaries (10-40-100 Gbps lambdas)

Optical mesh

High scalability (terabits/fiber) OXC, OXC+DCS, R-OADM Integrated EDFA (pre, in-line) GMPLS control (OSPF-TE, RSVP-TE)

Advanced Cyber-Infrastructure Laboratory

Wavelength Division Multiplexing

Single Mode Fiber (SMF) ITU-T Spectrum
5 Fiber Loss (dB/km) 1.0 0.5 0.2

O-Band
Legacy TDM, Ethernet band

E-Band
SMF water-peak profile

S-Band

DWDM (0.8 nm, 0.4 nm)

C & L-Bands

Improved lowwater-peak fibers

1300 nm
IP/Eth SONET

1400 nm

1500 nm

1600 nm

DWDM Transmission (Mid-1990s) SMF fiber

EDFA

Multiple colors per fiber: Frequency division mux (FDM) Massive scalability (terabits): 100+ s (2.5, 10 Gb/s each) Full-band amplifiers (EDFA): No costly per-ch regeneration Transparent bypass features: Optical cross connect (OXC)

SAN Cable
Laser transponders

Mux / demux

Advanced Cyber-Infrastructure Laboratory

Optical Wavelength Routing

Optical Cross Connect (OXC)
No electronic conversion (transparent)

Automate provisioning of wavelengths

Control Fabric Demux Mux

Rapid progress in architectures: GMPLS, UNI, OTN, ASTN, etc

Extensive research developments:

Switching designs, RWA, survivability, network design, etc

Multi-Hop Lightpath Routing

Client Router A Optical bypass (no electronics), much lower cost/complexity Client Router B

Optical Network

Advanced Cyber-Infrastructure Laboratory

Outline
Background Key Developments Scheduled Services Design Ongoing Efforts Conclusions

Advanced Cyber-Infrastructure Laboratory

E-Science Applications
High energy physics, fusion, astrophysics
Experimentation,

computation generate massive datasets Need to transfer, visualize, remotely steer sensors Large Hadron Collider, Spallation Neutron Source, Advanced Light Source, Terascale Supernova Initiative, etc

Some Facts
     Petabytes-exabytes dataset sizes (5 yrs)
Bio-informatics, Gene expression, sequence analysis genomics Deterministic transfer requirements Terabyte-level datasets common Need to transfer, visualize, steer Dedicated bandwidth & low jitter (QoS) Joint Genome Institute

Data backhauling, router-to-router TE Scheduled demands/work-flows

Climate & geographical change modeling connections Very massive datasets Visualization component
Long/short-term

Advanced Cyber-Infrastructure Laboratory

Key NREN Infrastructures

Internet2 DOE Energy Sciences Net (ESNet) DOE UltraScience Net (USN)

Some Facts

National Lambda Rail

Blend networks/computing/storage: Grid-computing, workflow paradigms Many technologies deployed (Gbps-Tbps): IP, MPLS, Ethernet, DWDM, SONET/SDH Increased inter-domain peerings to support wider collaborations Union European Interconnection with state RENs (Quilt)
CANARIE (Canada)

Global Lambda Integrated Facility

Abilene

Global Ring Network for Adv. Applications Development

Advanced Cyber-Infrastructure Laboratory

Demand Growth
Traffic Increase 10x Every 4 years

2005

2014

Advanced Cyber-Infrastructure Laboratory

10

Emerging Service Requirements

Immediate on-demand reservation
Generally suited for most mid-sized transfers (gigabytes) Already implemented in most networks (k-SP heuristics) Extensive standards support (routing, signaling, PCE)

Further need for network connection scheduling

Massive transfers preclude on-demand reservations Scientific workflows/apps need timing of network connections New advance reservation (AR) services stagger users:
I want connection service from A-B from time tstart to tend

Many commercial applications as well:

Special broadcasts, video conferencing, storage backups, etc

Advanced Cyber-Infrastructure Laboratory

11

Outline
Background Key Developments Scheduled Services Design Ongoing Efforts Conclusions

Advanced Cyber-Infrastructure Laboratory

12

Overview & Challenges

Prior work in AR
Various AR scheduling algorithms proposed (IP, DWDM) Largely heuristic (graph-theoretic) schemes, some ILP: Maintain link BW-timeline windows, search for feasible routes Many variants as well, e.g., variable start/stop, data sizes, etc

Open areas
Rerouting of future reservations:
Very few studies to date, no optimization work either

Real-world network implementation:

Existing focus mostly on idealized algorithm design

Advanced Cyber-Infrastructure Laboratory

13

AR Rerouting Strategies
Overview of approach
Reroute non-active future reservations (non-disruptive) Tradeoff overhead (complexity) vs optimality (effectiveness) Existing studies: try to min. # rerouted paths Open issues:
- Path selection (new request) to minimize disruptions - Proper resource re-distribution, i.e., prevent future call rejections

Our contributions
Integer linear optimization (ILP) formulation for rerouting Graph-based rerouting heuristics:
- Apply load-balancing concepts (extend gains in IR settings) - Candidate path concept to streamline rerouting selection

Advanced Cyber-Infrastructure Laboratory

14

AR Rerouting Strategies
Exponential time, O(2N)

Relative Run-Time Complexity

Exponential O(2n), n<<N Polynomial time, O(Nk)

No re-routing

Graph-based heuristics

Dynamic ILP (partial rerouting)

Global ILP rerouting

Solution Algorithms

Advanced Cyber-Infrastructure Laboratory

15

ILP Optimization Formulations

Global, idealized settings
Assume demands known a-priori (real-world are sequential) Use to bound performance, various AR-related studies:
- Assume full visibility (all connections inactive, schedulable) - Segment timeline into fixed timeslots for arrival/departure

AR rerouting considerations
Timeline dimension induces huge complexities
E.g., 4-node mesh, 30 reqs, 50 timeslots 16x30x50=24k variables

Very lengthy compute times (global ILP formulation) Propose a dynamic ILP adaptation

Advanced Cyber-Infrastructure Laboratory

16

Dynamic ILP (DILP)

Motivations
Customize ILP to improve scalability Preclude full a-priori demands, i.e., sequential on-demand Integrate w. simulation, i.e., OPNET Modeler TM + lp_solve

Key methodologies
Run shortened ILP for each incoming request w. inputs:
- BW timelines for all links (from current time) - Pending and active reservation lists

Limit timeslots by choosing farthest look-ahead time:

I.e., based on reservation w. latest ending time

Drastically reduce variable counts (compute time):

E.g., 4-node mesh w. 3 pending reservations over 10 time slots 4x4x(3+1)x10=640 variables
Advanced Cyber-Infrastructure Laboratory 17

Notational Overview
Key variables
V: E: rn : ri: T: Rat: Rpt: Tl: pn,e,t: v e: e v: Set of network nodes Set of network links Reservation n, denoted by 5-tuple: {source sn, dest dn, start time tsn, end time ten, bandwidth bn } Incoming reservation Note Current timeslot (when ILP triggered) This is a generic formulation for partial rerouting Set of active reservations at time t Set ofeasily generalize forat time t Can pending reservations full global ILP rerouting: Max. look-ahead time, i.e., only re-optimizeRatin [T , Tl] - Initialize to unloaded network, i.e., rn = { } - if reservation rn look-ahead time Tl t; 0 not 1Set maximum used link e at timeslot= maxif{ tsn } If node v is the egress node of link e If node v is the ingress node of link e

Objective
Minimize the total resource assigned to current pending & incoming reservations, i.e., (bandwidth x path length product)
minimize

r nRT {r i p

b
} eE T <t eTl

p n,e,t

Advanced Cyber-Infrastructure Laboratory

18

Conditional Expressions
minimize
rn

n,e,t

p n,e,t

RT {r i } eE T <t eTl p

s.t.

p
sn

= 1 r n R T {r i },t sn e t e ten p = 0 r n RT {r i },t sn e t e ten p = 1 r n R T {r i },t sn e t e ten p

= 0 r n RT {r i }, t sn e t e ten p
r n R T {r i },t sn e t e ten , v {s n , d n } p
e E,T e t e Tl

: Outbound traffic at source node : Inbound traffic at source node

pe
n,e,t

p
e ps n

p
e pd n

n,e,t

: Inbound traffic at destination node : Outbound traffic at destination node : Transit traffic at intermediate node : Total bandwidth at each time slot : Consistent path during the time span

p
dn

n,e,t

pe
n,e,t

p
e pv
rn

= p n,e,t
v pe
n i

b
T a T p

p n,e,t e W

R R {r }
T r n Ra RT {r i }, e E,t sn e t < ten p

p n,e,t = p n,e,t+1

Advanced Cyber-Infrastructure Laboratory

19

AR Rerouting Heuristic
Two-Stage Heuristic Solution
AR Request R = {src, dest, x, T1, T2}

AR load-balancing algorithm No feasible paths

Setup success Feasible path found

AR load-balancing rerouting

Setup success Feasible paths found for input request and all rerouted connections

Setup failure Insufficient resources

C. Xie, et al, "Load-Balancing Connection Rerouting in Advance Reservation Networks", IEEE Communications Letters, June 2010.

Advanced Cyber-Infrastructure Laboratory

20

AR Load-Balancing Heuristic LoadAR Request 5-Tuple R = {src, dest, x, T1, T2} src: dest: x: T1: T2: Source node Dest node Req. capacity Start time Stop time 1) 2) 3) Regular AR (Non-Rerouting) Compute K shortest paths from src-dest over G(V , E), only consider feasible links, i.e., capacity x in [T1, T2] Compute bottleneck bandwidth of each path in desired interval [T1, T2] (min. BW of all path links) Select feasible path with max. bottleneck bandwidth
Link A-B Timeline

Incoming request: R = {A, E, 0.5X, T1, T2}

Path1

H is bottleneck bandwidth, Path1 feasible if H 0.5X

Path 1

t
A C G
Path2

T1

T2

time

X
F

Link B-E Timeline

T1

T3

T1

T2

time
21

Advanced Cyber-Infrastructure Laboratory

AR Load-Balancing Rerouting LoadOverall Goals Find capacity for a request by rerouting a minimum number of reservations Recover partial capacity, x ( 1) Use modified Dijkstras SP algorithms and previous AR LB (polynomial time)
3

Incoming request: R = {A, E, 0.5X, T1, T2} Let =0.5, hence R = {A, E, 0.25X, T1, T2} G(V , E) for interval [T1, T2]
2 B 1 G A 1 2 C 1 D 3 F 2 E 4

Rerouting Phase Let R = {src, dest, x, T1, T2} if (found load-balancing path for R ) - Build G(V , E) where weight of a link is the min. # of resv. to exceed (1- )x in [T1, T2] - Compute shortest src-dest path on G(V , E), i.e., candidate path that has smallest rerouting candidate set (RCS) - Try to re-establish all connections in RCS, success if all found (use temp. graph copy) else Reject request R

Candidate path A-C-D-E gives smallest RCS, i.e., 4 reservations Minimum service disruptions

Advanced Cyber-Infrastructure Laboratory

22

Performance Evaluation
OPNET ModelerTM Overview State-of-the-art tool, widely-used Complete GUI, ease of use Full C/C++ coding interface Integrated lp_solve ILP toolkit Hierarchical modeling: Subnet, node, link, process In-House Expertise Full development site: 10+ licenses, 30,000+ LOC Advanced protocols suite: IP, MPLS, Eth, SONET, optical Extensive topology database

Advanced Cyber-Infrastructure Laboratory

23

Simple 4-Node Network 4Topology Testcase Parameters 30 random connection requests 10 Gbps links speeds 0.2 - 1 Gbps requests (200 Mbps increments) Exp. inter-arrival / hold / book-ahead times

0.2
Min-hop (no rerouting) Load-balancing Rerouting

Analysis Results

0.15

DILP GILP

BBR (%)

Scheme
LB Heuristic Dynamic ILP

Run Time
30 sec 45 sec 3 days

0.1

0.05

Global ILP

-4.44E-16 9 10 11 12 13 14 15 16 17

Load (Erlang)
Advanced Cyber-Infrastructure Laboratory 24

8-Node Network
Topology Testcase Parameters 1,000 random connection requests 10 Gbps links speeds 0.2 - 1 Gbps requests (200 Mbps increments) Exp. inter-arrival / hold / book-ahead times

0.1

Analysis Results

BBR (%)

Scheme
0.01
Min-hop LB-R DILP

Run Time
2 min 6 min Unbounded

LB Heuristic Dynamic ILP Global ILP

0.001 25 35 45 55 65

Load (Erlang)
Advanced Cyber-Infrastructure Laboratory 25

1616-Node NSFNET
Topology Testcase Parameters 10 Gbps link speeds 0.2-1 Gbps requests (200 Mbps increments) Exp. inter-arrival / hold/ book-ahead times Heuristics only, ILP schemes do not converge Analysis Results
0.1 HR LR LB-R =0.5

20-80% higher carried load/revenues

HR

Blocking
Bandwidth Utilization(%)

0.44

Resource Utilization

0.42

LR LB-R =0.5

0.4

BBR(%)

0.01

0.38

0.36

0.34

0.001 50 70 90 110 130 150 170

0.32 50 70 90 110 130 150 170

Load(Erlang)

Load(Erlang)

Advanced Cyber-Infrastructure Laboratory

26

2727-Node Deutsche Telekom

Topology Testcase Parameters 10 Gbps link speeds 0.2-1 Gbps requests (200 Mbps increments) Exp. inter-arrival / hold/ book-ahead times Heuristics only, ILP schemes do not converge Add HC-R heuristic Analysis Results
0.1

Blocking
3.6 3.5

Resource Utilization
min-hop LR
3.4 3.3 3.2 3.1 3 2.9 200

0.01 BBR (% )

Mean Path Length(Hop)

HC-R LB-R =0.5

min-hop LR HC-R LB-R =0.5

0.001

0.0001 180

230

280

330

380

430

480

530

580

250

300

350 400 Load(Erlang)

450

500

550

Load (Erlang)

Advanced Cyber-Infrastructure Laboratory

27

Overhead Comparisons
Measure average time between rerouting events Compare proposed LB rerouting with hop count (HC) rerouting Analysis Results
NSFNET
350
Time Units per rerouted connection
Time Units per rerouted connection

Deutsche Telekom
200 180 160 (1000 time units) 140 120 100 80 60 40 20 0 200 250 300 350 400 450 500 550

300 (1000 time units) 250 200 150 100 50 0 50 70 90 110

HC-R LB-R =0.5

HC-R LB-R =0.5

130

150

170

Load(Erlang)

Load (Erlang)

Advanced Cyber-Infrastructure Laboratory

28

Outline
Background Key Developments Scheduled Services Design Ongoing Efforts Conclusions

Advanced Cyber-Infrastructure Laboratory

29

RealReal-World Challenges
Current implementation status
AR schemes require global timelines for all links:
Real world distributed control, multiple domains

Existing protocols for IR only (GMPLS):

Routing, signaling, path computation, etc

Few centralized/proprietary AR solutions:

ESNet OSCARS, EU C RISM testbed

Our contributions
First complete framework for distributed AR (IEEE Globecom10) Augment existing protocols w. timeline state (OSPF-TE) Further efforts on multi-domain AR

Advanced Cyber-Infrastructure Laboratory

30

Related Challenges
Domain 2 Domain 6 Destination

Domain 1 Source Domain 4 Domain 5 Domain 3

Overview Real-world networks are complex, decentralized A single entity cannot maintain global state: Scalability, privacy reasons Some standards emerging, key research area: On demand IR operation only Propose distributed AR solutions, limited visibility
Advanced Cyber-Infrastructure Laboratory 31

Proposed Approach
Extend GMPLS Standards

Physical topologies and resources

Routing policies, database MIB, timers

Skeleton path scheduling algorithms & policies

Topology Abstraction Condensed domain state

Hierarchical Inter-Domain Routing Global abstract views

AR Scheduling& Signaling Setup

Inter-domain AR routes & schedules

Advanced Cyber-Infrastructure Laboratory

32

Topology Abstraction
Objectives
Optimize externally-divulged (timeline) state

Various renditions: simple node, mesh, star, bus, etc Well-studied for IR provisioning (IP, DWDM) Proposed first extension for AR settings
Full Domain Topology
G(V,E)

Full Mesh Border nodes revealed

Simple Node Max. summarization

Advanced Cyber-Infrastructure Laboratory

33

Conclusions
Many cyber-infrastructure advances
Maturation of key technologies (Layers 1-3) Focus now on expanding services & applications Many directions to build upon

Some crucial open research problems

Network scheduling: benefits from load-balancing, rerouting Need to buid real-world proof-of-concept test-beds Many avenues for collaboration, development, training

Advanced Cyber-Infrastructure Laboratory

34

Thank You!

Question & Answer

Advanced Cyber-Infrastructure Laboratory

35