Grid Computing Fall 2006 8/29/2006

Paul A. Farrell

Course Web Site

• Web Site
Grid Computing – http://www.cs.kent.edu/~farrell/grid06/

Paul A. Farrell • Syllabus

Fall 2006 – http://www.cs.kent.edu/~farrell/grid06/syllabus.html
• Lectures
– http://www.cs.kent.edu/~farrell/grid06/lectures/index.html
• References
Based on
The Grid: Core Technologies – http://www.cs.kent.edu/~farrell/grid06/reference/index.html
Maozhen Li, Mark Baker
John Wiley & Sons; 2005, ISBN 0-470-09417-6
Slides from Amy Apon, University of Arkansas
Slides from

Paul A. Farrell 2006 Grid Computing 1 Paul A. Farrell 2006 Grid Computing 2

Your Background? Distributed Applications and Middleware

• Name • Application runs “on top” of middleware that unifies
• MS or PhD student, years at Kent the resources into a single coherent virtual machine
• CS background Parallel Applications
Parallel Applications
Parallel Applications
Sequential Applications
– Programming: OOP, C++, Java Sequential Applications
Sequential Applications Parallel Programming Environment
– Operating Systems: courses, used Unix
– Networking: any courses Cluster Middleware
(Single System Image and Availability Infrastructure)
– Cluster Computing, Parallel & Distributed Computing
PC/Workstation PC/Workstation PC/Workstation PC/Workstation
• What interests you in this topic?
Communications Communications Communications Communications
Software Software Software Software

Network Interface Network Interface Network Interface Network Interface

Hardware Hardware Hardware Hardware

Cluster Interconnection Network/Switch

Paul A. Farrell 2006 Grid Computing 3 Paul A. Farrell 2006 Grid Computing 4

Dept of Computer Science

Kent State University 1
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Distributed Systems History Distributed Systems to Grids

D-Grid
Bespoke pioneering distributed systems, e.g. SABRE & SAGE Bespoke pioneering distributed systems
starts
Linklider proposes shared multi-site computing Many
Linklater proposes shared researchcomputing
multi-site grids
ARPA net Using many protocols & M/W stacks
IBM CICS Web Services
IBM CICS
Ethernet ARPA net EDG EGEE
TCP EGEE II
Unicore networks
Dozens of academic networks Dozens of academic
Many incompatible protocols Two dominant protocols
Globus
CORBA & DCOM CORBAStarts
& DCOM
Collaboration via shared
IP-based Internet bio/chem/medical IP-based Internet
I-way
Academic & Research “DBs” Academic & Research
WWW Condor WWW

1960 Paul A. Farrell

1970 2006 1980 1990Grid Computing
20005 1960 Paul A. Farrell
1970 2006 1980 1990Grid Computing
20006

A Grid Computing Timeline Precursors of the Grid

8
‘9 G
F
5 S C ms G
‘9 u • Parallel computing in the 1980’s and ’90’s
g at or or
m
tin s F ,f
– Focused on providing powerful mechanisms for managing communication
pu rm rid e between processors, and development and execution environments for parallel
m fo G er
g r ed
Co um AP m er
a pe orm machines
r r & s p p f
pe Fo m pa y” G .0 G A PVM, MPI, HPF, and OpenMP for scalable applications
: Su r i d ean oru my” log -W
o
v1 E F • The first modern Grid is the I-WAY (SC95)
s SA & G
ay G rop F to i
SA
rid Ana Phy G G GF O – Aggregate a national distributed testbed with over 17 sites networked together
I-W US E u G “ “ O O G rm by the vBNS
1995 ’96 ’97 ’98 ’99 2000 ’01 ’02 ’03 ’04 ’05 fo
2006 – Over 60 applications were developed for the conference
– A rudimentary Grid software infrastructure to provide access, enforce security,
Japan government funds: coordinate resources, and other activities
• Business Grid project – Distributed computing : geographical separation
• UK e-Science programme starts
• NAREGI project
– Grid research : integration and management of software
• DARPA funds Globus Toolkit & Legion
• EU funds UNICORE project Today:
• US DoE pioneers grids for scientific research • Grid solutions are common for HPC
• Grid-based business solutions are
• NSF funds National Technology Grid becoming common
• NASA starts Information Power Grid • Required technologies & standards are
evolving
Source:Paul A. Farrell
Hiro Kishimoto 2006May 2006
GGF17 Keynote
Grid Computing 7 Paul A. Farrell 2006 Grid Computing 8

Dept of Computer Science

Kent State University 2
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

On the Road to the Grid What is a Grid?

A grid is a system consisting of
• Projects after I-WAY – Distributed but connected resources and
– The Globus and Legion infrastructure – Software and/or hardware that provides and manages logically seamless
access to those resources to meet desired objectives
– Condor experimented with high-throughput scheduling
– Mars and Prophet experimented with high-performance scheduling
– NWS (Network Weather Service) focused on resource monitoring and prediction
– Storage Resource Broker focused on uniform access to heterogeneous data
resources License
Web
– NetSolve and Ninf focused on remote computation via a client-server model server
• The Grid Forum in the late 1990’s
– Open Grid Services Architecture (OGSA) which integrates Globus and Web Handheld Server Supercomputer
Services approaches
Workstation Cluster Data Center

Database Printer
R2AD

Paul A. Farrell 2006 Grid Computing 9 Source:Paul A. Farrell

Hiro Kishimoto 2006May 2006
GGF17 Keynote
Grid Computing 11

What is the Grid? Early Grid Example

• Type of distributed system
• But what are special characteristics?
• First definition
– Ian Foster and Carl Kesselman The Grid: Blueprint for a
New Computing Infrastructure [1]
• “A computational grid is a hardware and software
infrastructure that provides dependable, consistent,
pervasive, and inexpensive access to high-end
computational capabilities."
– Based on original linking of supercomputer centers

Paul A. Farrell 2006 Grid Computing 12 Paul A. Farrell 2006 Grid Computing 13

Dept of Computer Science

Kent State University 3
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Later definitions Foster Checklist – Is it a Grid?

• “A grid is a software framework providing layers of services to • Coordinated resource sharing with no centralised
access and manage distributed hardware and software
resources” point of control
– CCA, http://www.extreme.indiana.edu/ccat/glossary.html
• Users reside within different administrative domains
• “… widely distributed network of high-performance computers,
stored data, instruments, and collaboration environments shared • Use of standard, open, general-purpose protocols
across institutional boundaries”
– IPG, http://www.ipg.nasa.gov/ipgflat/aboutipg/glossary.html
and interfaces
• Flexible, secure, coordinated resource sharing among dynamic • Delivers non-trivial qualities of service
collections of individuals, institutions, and resource
– components can be used in a co-ordinated way to deliver
– “The Anatomy of the Grid: Enabling Scalable Virtual
Organizations” combined services, which are appreciably greater than sum
• "coordinated resource sharing and problem solving in dynamic, of the individual components
multi-institutional virtual organizations" • http://www.gridtoday.com/02/0722/100136.html
– Foster, Kesselman and Tuecke 2001
– Most commonly used currently

Paul A. Farrell 2006 Grid Computing 14 Paul A. Farrell 2006 Grid Computing 15

Grid & Related Paradigms What is a Grid? - Summary

Cluster • Grid is about conditional sharing of resources,
Distributed Computing
• Tightly coupled
• Loosely coupled
• Homogeneous including computers, storage, sensors, and networks
• Heterogeneous
• Cooperative working – Based on factors like trust, resource-based policies,
• Single Administration
negotiation
• Co-ordinated problem solving
– Beyond simple client-server computing
• Dynamic, multi-institutional virtual organisations
Grid Computing (VOs)
• Large scale – New communities that overlay classical organisation
• Cross-organizational
• Geographical distribution
structures
• Distributed Management

Utility Computing
• Computing “services”
• No knowledge of provider
Source:Paul A.• Farrell
Enabled
Hiro Kishimoto 2006
by gridMay
GGF17 Keynote technology
2006
Grid Computing 16 Paul A. Farrell 2006 Grid Computing 17

Dept of Computer Science

Kent State University 4
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Multi-disciplinary Simulations: Aviation Safety

Data Grids for High Energy Physics
Wing Models Image courtesy Harvey Newman, Caltech
~PBytes/sec
•Lift Capabilities 1 TIPS is approximately 25,000
•Drag Capabilities
Stabilizer Models Online System ~100 MBytes/sec SpecInt95 equivalents
Airframe Models •Responsiveness
Offline Processor Farm
There is a “bunch crossing” every 25 nsecs.
~20 TIPS
There are 100 “triggers” per second
~100 MBytes/sec
Each triggered event is ~1 MByte in size
•Deflection capabilities
•Responsiveness ~622 Mbits/sec
Tier 0 CERN Computer Centre
or Air Freight (deprecated)
Crew Capabilities
- accuracy Tier 1
- perception France Regional Germany Regional Italy Regional FermiLab ~4 TIPS
Centre Centre Centre
- stamina
- re-action times ~622 Mbits/sec
- SOP’s
Engine Models
Tier 2 Caltech Tier2 Centre
Tier2 Centre
Tier2 Centre
Tier2 Centre
~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS
~622 Mbits/sec

•Braking performance Institute

Human Models Institute Institute Institute
•Steering capabilities
•Thrust performance ~0.25TIPS Physicists work on analysis “channels”.
•Reverse Thrust performance
•Traction Each institute will have ~10 physicists working on one or more
Physics data cache
•Dampening capabilities
•Responsiveness ~1 MBytes/sec channels; data for these channels should be cached by the
•Fuel Consumption institute server

Source NASA Landing Gear Models Tier 4

Whole system simulations are produced by coupling all of the sub-system simulations Physicist workstations

Paul A. Farrell 2006 Grid Computing 18 Paul A. Farrell 2006 Grid Computing 19

BIRN Network Grid Related Standards Bodies

• Global Grid Forum (GGF)
– primary standards setting organisation for the Grid
• OASIS
– a not-for-profit consortium that drives the development,
convergence, and adoption of e-business standards
• Related Standards
– Distributed Management Task Force (DMTF)
• Common Information Model (CIM)
• Web-Based Enterprise Management (WBEM) [14]
• World Wide Web Consortium (W3C) Web Services
standards, particularly those that relate to XML.
IT Infrastructure to hasten the derivation of new understanding and
treatment of disease through use of distributed knowledge

Paul A. Farrell 2006 Paul A. Farrell 2006 Grid Computing 21

Dept of Computer Science

Kent State University 5
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Global Grid Forum Document Types Global Grid Forum Document Types
• Informational • Community Practice
– useful idea or set of ideas – common practice or process, with the objective to influence
the community
• GFD.7 (A Grid Monitoring Architecture)
– GFD.1 (GGF Document Series)
• GFD.8, (A Simple Case Study of a Grid Performance
– GFD.3 (GGF Management)
System)
– GFD.16 (GGF Certificate Policy Model
• GFD.11 (Grid Scheduling Dictionary of Terms and
Keywords). • Recommendations
– document a specification, analogous to an Internet
• Experimental Standards track document
– inform the community about a useful experiment, testbed, or • GFD.15 (Open Grid Services Infrastructure)
implementation of an idea of set of ideas • GFD.20 (GridFTP: Protocol Extensions to FTP for the
• GFD.5 (Advanced Reservation API) Grid)
• GFD.21, (GridFTP Protocol Improvements) • GFD.23 (A Hierarchy of Network Performance
• GFD.24 (GSS-API Extensions) Characteristics for Grid Applications and Services)

Paul A. Farrell 2006 Grid Computing 22 Paul A. Farrell 2006 Grid Computing 23

Grid Architecture A Community Grid Model

• Open Grid Services Architecture (OGSA) - GGF 2002 • A layered abstraction of the Grid
– Informational specification that aims to define a common, standard,
and open architecture for grid-based applications
– Goal is to standardise almost all the services that a grid application
may use, for example, job and resource management services,
communications and security
– A service-oriented architecture (SOA) for the Grid that realises a
model of a computing system as a set of distributed computing
patterns realised using Web Services as the underlying technology
– Defines service interfaces and identifies the protocols for invoking
these services
– Currently the OGSA document does not contain sufficient in
formation to develop an actual implementation of OSGA-based
system

Paul A. Farrell 2006 Grid Computing 24 Paul A. Farrell 2006 Grid Computing 25

Dept of Computer Science

Kent State University 6
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Web Services-based standards and

A Community Grid Model
specifications
• Global Resources – Program-to-program interaction (SOAP, WSDL, and UDDI);
– Such resources include computers, networks, data archives, instruments, – Data sharing (eXtensible Markup Language - XML);
visualization devices, etc.
– Be distributed, heterogeneous, very different performance, and highly dynamic
– Messaging (SOAP, and WS-Addressing)
• Common infrastructure – Reliable Messaging (WS-Reliable Messaging);
– The software services which will represent the Grid as a unified virtual platform and – Managing workload (WS-Management);
provide the target for more focused software and applications
– Example : NFS’s Middleware Initiative (NMI), OGSA – Transaction-handling (WS-Coordination, and WS-
• User-focused grid middleware, tools, and services AtomicTransaction),
– To enable applications to use Grid resources by masking some of the complexity – Managing resources (WS-RF or Web Services Resource
involved in system activities such as authentication, file transfer, etc.
Framework);
– To connect applications and users with the common Grid infrastructure
– Establishing Security (WS-Security, WS-
SecureConversation, WS-Trust, and WS-Federation),
– Handling metadata (WSDL, UDDI, and WS-Policy),

Paul A. Farrell 2006 Grid Computing 26 Paul A. Farrell 2006 Grid Computing 27

Web Services-based standards and

specifications
Some topics for later in the course
– Building and integrating Web services architecture over a • System Infrastructure
grid (see OGSA), – RPC, RMI, SOAP, WSDL, OGSA, Globus, WSRF
– Overlaying business process flow (Business Process – Semantic Grid, Autonomic Computing
Engineering Language for Web services - BPEL4WS),
– Triggering process flow events (WS-Notification). • Basic Services
– Grid Security, Grid Monitoring
• Job Management and User Interaction
– Grid Scheduling and Resource Management
– Workflow Management
– Grid Portals
• Applications

Paul A. Farrell 2006 Grid Computing 28 Paul A. Farrell 2006 Grid Computing 29

Dept of Computer Science

Kent State University 7
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

RPC and RMI Grid Architecture

• Remote Procedure Call (RPC)
• Extends the client/server concept by making the
definition of an interface more formal, clarifying rules
for parameter passing, adding capabilities for locating
services, etc.
• RMI (Remote Method Invocation) is RPC applied to
object. We will study Java RMI
• Want to be able to share among any potential participants
• Interoperability in the key issue
NOTE: there is a change from the “Anatomy” paper that you will
read – the paper says that the Grid architecture is a protocol
architecture. The newer approach is a service oriented
architecture

Paul A. Farrell 2006 Grid Computing 30 Paul A. Farrell 2006 Grid Computing 31

Components for authentication in virtual

organization environments
Web Services
• Single sign on – access to multiple resources after logging in • A client/server system
just once
• Delegation – a program can access the resources that a user is
authorized to access • Concept similar to Remote Procedure Call (RPC),
• Integration with various local security solutions – for example, Remote Method Invocation (RMI), only applied over
local Unix security HTTP
• Trust relationships – between individual users and resources;
between resources and user groups; between virtual and real
organizations

Paul A. Farrell 2006 Grid Computing 32 Paul A. Farrell 2006 Grid Computing 33

Dept of Computer Science

Kent State University 8
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Web Services Web Services Architecture

The Web Services

Architecture is specified
and standardized by the
World Wide Web
Consortium, the same
organization responsible
for XML, HTML, CSS,
etc.

Paul A. Farrell 2006 Grid Computing 34 Paul A. Farrell 2006 Grid Computing 35

Web Services missing features Grid Services

• Common interface specification supports the interoperability of
• At the time the OGSI V1.0 spec was published there was a gap discrete, independently developed services
between the need to define stateful Web Services and what
was provided by the latest version of Web Services in WSDL • Based on extensions of Web Services
1.1 – Web Services were stateless and non-transient • OGSA – Open Grid Services Architecture
• OGSI – Open Grid Services Infrastructure
• The result was the definition in OGSI of Service Data – a • GT3 – Globus Toolkit version 3, an implementation of OGSI
common mechanism to expose a service instance’s state data
for query, update, and change notification
• Also, Grid Services uses a Factory to manage instances – to
allow transient and private instances

Paul A. Farrell 2006 Grid Computing 36 Paul A. Farrell 2006 Grid Computing 37

Dept of Computer Science

Kent State University 9
Grid Computing Fall 2006
Paul A. Farrell
Paul A. Farrell 2006 Grid Computing 38
Higher Level Globus Toolkit Services
Data Services include
• Replica Management
Base Services include
• Managed Job Service
• Index Service
• Reliable FTP
Many documents define
GT3 Security Services
Paul A. Farrell 2006 Grid Computing 40
Dept of Computer Science
Kent State University
8/29/2006
Grid Services Factory
Paul A. Farrell 2006 Grid Computing 39
OGSI is a standard for Grid computing
• There are many advantages to working within a
standard framework
– Single sign-on
– Remote deployment of executables
– Computation management, data movement
– Benefits of working with an international community of
developers and users
– A framework enables the definition of higher-level services
Paul A. Farrell 2006 Grid Computing 41
10
Grid Computing Fall 2006 8/29/2006
Paul A. Farrell

Recent Changes to Grid Standards

• Introduction of Web Services Resource Framework
(WSRF), January, 2004
– Web services vendors recognized the importance of OGSI
concept but would not adopt OGSI as it was defined
(summer 2003)
– Globus Alliance teamed up with Web services architects and
came up with WSRF
– Add the ability to create, address, inspect, discover, and
manage stateful resources

Paul A. Farrell 2006 Grid Computing 42

Dept of Computer Science

Kent State University 11