Вы находитесь на странице: 1из 136

Grid Computing

Richa Sinha
Electrical Power Grid Analog
Electrical power grid The Grid
users (or electrical appliances) get users (or client applications) gain
access to electricity through wall access to computing resources
sockets with no care or (processors, storage, data,
consideration for where or how applications, and so on) as needed
the electricity is actually with little or no knowledge of where
generated. those resources are located or what
the underlying technologies,
The power grid links together hardware, operating system, and so
power plants of many different on are
kinds
"the Grid" links together computing
resources (PCs, workstations,
servers, storage elements) and
provides the mechanism needed to
access them.
Grid Computing Introduced
Idea of grid was brought by Ian Foster, Carl Kesselman and Steve
Tuecke in the year 1970
Subset of distributed computing
Internet=network of communication
Grid computing=network of computation
History
Early 1980-1990 Concept of Parallel Computing using
PVM
MPI etc

GRID can be thought as a tool extending parallel computer


paradigms from a cluster to a distributed environment which is
loosely coupled.
Grid links resources like
Storage
Computation
Visualization
Instruments
Grid Computing is About

Dynamically linking of
resources
Ensembling
For execution of large
scale, resource
intensive, distributive
applications
Grid Computing Defined as
It is method of harassing the power of many computers in a
network to solve the problem requires large number of processing
cycle and requires huge amount of data
Global Grid Forum (1990)
OGSA (Open Grid Services Architecture) formed by this forum
defined service of Grid
System Management and automation
Workload/Performance
Security
Availability/Service Management
Logical Resource management
Clustering Service
Connecting Issues
Physical Storage Management
Why need of Grid Computing?
Core networking technology now accelerates at a much faster rate
than advances in microprocessor speeds
Exploiting under utilized resources
Parallel CPU capacity
Virtual resources and virtual organizations for collaboration
Access to additional resources
Resource Balancing
Reliability
Management
Types of Grid

Information Grid (Web Service Grid


HTML, file Sharing) (OGSA,SOAP,WSDL, UDDI)

Resource Grid(Data Grid and Computational Grid)


(Network Storage Computing)

Access Storage
Data Grid and Computation Grid
A data grid is a grid computing A computational grid is a hardware
system that deals with data and software infrastructure that
the controlled sharing and provides dependable, consistent,
management of large amounts pervasive, and inexpensive access to
of distributed data high-end computational capabilities
Mainly used in broad areas Data
Mining Mainly used all broad areas
Main Operations
Main Operations
Are Cluster of Clusters, CPU scavenging,
Access, Transfer and Modify massive Provide Computation power, On demand
datasets stored in distributed storage Resources
Application of Grid
Life science- Bio Informatics, genomics, Neuroscience
Engineering: NASA, IPG
Data Oriented: GEODISE aircraft engineering design GRID,
DAME grid to manage data from aircraft engine sensors
Physical science : Particle physics data GRID, INFN(Italian
National Institute for Research in Nuclear and subnuclear Physics)
Astronomical Data Grid
Grid Topologies
Intragrid
Local grid within an organization
Trust based on personal contracts
Extragrid
Resources of a consortium of organizations
connected through a (Virtual) Private Network
Trust based on Business to Business contracts
Intergrid
Global sharing of resources through the internet
Trust based on certification
Types of Resources

Computation
Storage
Communications
Software and licenses
Special equipment, capacities, architectures, and policies
A Job Scheduler on Grid
Overview: Grid Computing Environment
An integrated set of tools
that extend the users
computing environment in
order to provide access to
Grid Services
Parts of GCE
Client
Portal
Services
Grid
Commodity
Grid Computing Environment

Grid Information Service


Grid Resource Broker

R2 Application
database
2 R3
R4

R5
RN
Grid Resource Broker
R6
R1
Resource Broker

Grid Information Service


Layered
Architecture an
GCE
Client Layer
Application Layer
Application services layer
Grid services
Resources layer
How does Grid Computing Environment Works..
Grid Information Grid Information Service
Service system collects Details of Grid resources
the details of the
available Grid
resources and passes
the information to the
resource broker.
Computational
jobs
Grid application
Processed jobs
Computation
result
User Resource Broker
A User sends computation A Resource Broker distribute the
or data intensive
application to Global
jobs in an application to the Grid Grid Resources
resources based on users QoS Grid Resources (Cluster, PC,
Grids in order to speed requirements and details of Supercomputer, database,
up the execution of the available Grid resources for instruments, etc.) in the
application. further executions. Global Grid execute the user
jobs.
Grid Infrastructure /Middleware
Middleware Cont
Many organizations involved
ex: Global Grid Forum, Globus, Legion, Condor, CoG Kit etc..

Components User-level
Security &
Secure communications (SSL) Core
Distributed security infrastructure
ASPECTS
Manage user credentials to selecting appropriate resources

Data Management
Transferring data throughout the grid and to users
GridFTP
Deals with high-performance, security and reliability

Information Services
Information DB about resources
Availability, capabilities,
Resource Management
Resource discovery, inventories, provisioning, monitoring, fault isolation,
autonomic capabilities
Grid Application Needed
Application partitioning for parallel Processing
Discovery and scheduling of task workflow
Data Communication when and where
Provisioning and distributed application
Result Management
Autonomic Features such as self- configuration, self-
optimization, self- recovery and self- management
GRID Application & Usage Pattern (Resource
Management)
Schedulers
Resource Broker
Load Balancing
Distribution of Workload among resources in GCE
Partitioning of jobs, indentifying the resources, queuing of the jobs

Grid Portals
Provide Uniform Access to GRID resources
Resource authentication, remote resource access, scheduling capabilities, monitoring status
information

Integrated Solutions
Combination of existing advanced middleware & application functionalities to provide more coherent &
high performance result across GCE
Schedulers
Schedulers are used in

Advanced resource reservation

SLA validation and


enforcement

Job and resource policy


management

Monitoring Job

Rescheduling and corrective


action for partial failover
Resource Broker

Provide Pairing service between service requester and service provider


Allocating proper resource or combination for task execution
Supporting users deadline and budget constraints
Key Components
Portal/user interface
Key Components
Security
Grid Security Infrastructure (GSI)
Key Components
Broker
Monitoring and Discovery Service (MDS)
Key Components
Scheduler
Key Components
Data management
Grid Access to Secondary Storage (GASS)
Key Components
Job and resource management
Grid Resource Allocation Manager (GRAM)
Resource Sharing in GCE
Grid Protocol Architecture
Fabric Layer
Interface to local resources
Defines the resources that can be shared E.g.. computational
resources, data storage, networks, catalogs
A resource can be a Physical resource or a logical resource
A logical resource can be implemented by their own internal protocol
Basic capabilities associated with the integration of resources
provide an inquiry
provide appropriate resource management
Connectivity Layer
Manages communications
Defines core communication and authentication protocols
Authentication solution for Virtual Organizations:
Single sign on: any multiple entities in the grid fabric to be authenticated once
Delegation: ability to access a resource under the current user permissions
Integration with local resource specific security solutions
User-based trust relationships
Data Security
Resource Layer
Sharing of a single Resource
Controls the secure negotiation, initiation, monitoring, sharing of
operations across individual layer
Two primary classes of resource layer protocols
Information Protocols
Operational State of Single Resource configuration, usage policies, SLA
Management Protocols
Access to shared resource, operation on resource, policy, accounting and payment, state
of operation

Negotiating access to a shared resource


Performing operation on a resource & monitoring the status of operation
Collective Layer
coordinating multiple resources
Responsible for global resource management
Common collective services in a Grid Computing system
Discovery services
Co allocation, scheduling Services
Community accounting and Payment Services
Monitoring and Diagnostic Services
Software Discovery Services
Application Layer
User-Defined Grid Applications
User applications constructed by utilizing the services defined at
each lower level
Each layer in the Grid Architecture provides a set of APIs and
SDKs for the higher layers of integration
GRID Organization and their Roles
Grid Organizations are divided on the bases of
Organizations developing grid standards and best practices guidelines

Organizations developing Grid Computing toolkits, frameworks, and


middleware solutions

Organizations building and using grid-based solutions to solve their


computing, data and network requirements

Organizations working to adopt grid concepts into commercial products, via


utility computing, and Business On Demand computing
GRID Organization and their Roles
GRID Organization and their Roles
Evolution of Grid Computing
First Generation

Second Generation

Third Generation
Organizations developing grid standards
Global Grid Forum (GGF)
OASIS (Organization for the Advancement of Structured
Information Standards)
W3C (World Wide Web Consortium) (Web services, XML,
Semantic Web)
IETF (Internet Engineering Task Force)
DMTF (Distributed Management Task Force)
Global Grid Forum (GGF) Goal
Create grid specifications, architecture documents, and best
practice guidelines
Handle intellectual property policies
Provide a forum for information exchange and collaboration
Improve collaboration among the people involved with grid
research, grid framework builders, grid deployment, and grid users
Create best practice guidelines from the experience of the
technologies
Global Grid Forum (GGF) areas
Application and programming environments
Architecture
Data
Information systems and performance
Peer-to-peer: Desktop grids
Scheduling and resource management
Security
Evolution of Grid Computing Technologies
Evolution Of Grid Computing(1990)
FAFNER 66(Factoring via Network enabled recursion)
This is a set up to factorize RSA130 using Number Field Sieve (NFS) using computational
resources of web-servers. NFS is implemented in such a way that it does not require any
communication between nodes, and is such efficient that it can be run in a workstation
with 4Mb of memory. CGI scripts on server side are invoked by the contributors.

I-WAY
This is a project to integrate High-performance Computing device with existing High
bandwidth networks. It uses ATM(Asynchronous Transfer mode), supporting both TCP
and direct ATM.
It uses I-POP servers acting as a gateway to I-WAY. I-POP servers are UNIX
workstations possessing standard software environment.
Each site participating in I-WAY ran I-POP server. The I-POP server allowed uniform
access, authentication, resource reservation, process creation and communication
functions.
Evolution of Grid Computing: Second
Generation(1998)
Requirements for the data and computation infrastructure
Distributed object systems
Grid resource brokers and schedulers
Grid portals
Integrated systems
Peer-to-Peer computing
Second Generation Core Technologies are
FAFNER was forerunner of SETI@home and Distributed.NET
I-WAY for Globus and Legion
Globus
GSI3 = GSI + Alignment with WS-Security

Globus GT3 middleware

MDS (Monitoring & Discovery Service) combines data discovery mechanism with
Lightweight Directory Access Protocol (LDAP)
Globus
A collaboration of Argonne National Laboratorys Mathematics and
Computer Science Division, the University of Southern Californias
Information Sciences Institute, and the University of Chicago's
Distributed Systems Laboratory.
A project to develop the underlying technologies needed for the
construction of computational grids.
Focuses on execution environments for integrating widely-
distributed computational platforms, data resources, displays,
special instruments and so forth.
The Globus Toolkit
The Globus Resource Allocation Manager (GRAM)
Creates, monitors, and manages services.

Maps requests to local schedulers and computers.

The Grid Security Infrastructure (GSI)


Provides authentication services.

The Monitoring and Discovery Service (MDS)


Provides information about system status, including server configurations,
network status, and locations of replicated datasets, etc.

Global Access to Secondary Storage (GASS)


Provides data movement and access mechanisms that enable remote programs
to manipulate local data.
Legion
Legion
Everything is an object (Host, Binding etc).
Classes manage their own instances (meta-classes)
Users can provide their own classes
Core objects implement common services
Condor and Condor-G

NMI, Grid Physics Network (GriPhyN), International Virtual Data Grid Laboratory (iVDGL)
TerraGrid etc
Condor
Original goal: high-throughput computing
Harvest wasted CPU power from other machines
Can also be used on a dedicated cluster
Condor-G Condor interface to Globus resources
Condor
Provides many features of batch systems:
job queuing
scheduling policy
priority scheme
resource monitoring
resource management

Users submit their serial or parallel jobs


Condor places them into a queue
Scheduling and monitoring
Informs the user upon completion
Nimrod

Scheduling Algorithm
Cost Optimization
Time Optimization
Cost-time optimization
Conservative time strategy
NSF Middleware Initiative (NMI)
Grid Research Integration Deployment and Support
(GRIDS) Center
Globus toolkit, Condor-G, GSI-OpenSSH, Network Weather service, Grid
Packaging tools, GridConfig, MPICH-G2, MyProxy etc

Enterprise and Desktop Integration Technologies


(EDIT) Consortium
LDAP Operational ORCA Kollector(LOOK), Privilege and Role
Management Infrastructure Standards Validation (PERMIS), openSAMIL
etc
Organizations building and using grid solutions to solve
their computing, data and network requirements

United States Department of Energy: Science Grid


Scientific Computing

European Union: EUROGRID Project


Scientific and Industrial Communities
Bio Grid, Metro Grid, CAE Grid, HPC Research Grid

European Union: Data Grid Project


High Energy Physics, Biology and Medical Image Processing, Earth
Observations
TeraGrid
TeraGrid
NSF high-performance computing facility
Nine distributed sites, each with different capability , e.g.,
computation power, archiving facilities, visualization software
Applications may require more than one site
Data sizes on the order of gigabytes or terabytes
TeraGrid
Solution: Use GridFTP and RFT with front end command line tool
(tgcp)
Benefits of system:
Simple user interface
High performance data transfer capability
Ability to recover from both client and server software failures
Extensible configuration
NASA Information Power Grid
Resource Broker
Performance Prediction
Job Manager
Portability Manager
Framework for Control and Observation in Distributed Environments
(CODE)
Test and Monitoring Service
Dynamic Accounting System
CORBA-IPG Infrastructure
Commercial Organizations Building and
Using Grid-Based Solutions
Business on Demand by IBM
Utility Computing HP
N1 technology Sun Micro
Technology
Advance of SOA with Web Service
Hardware virtualizations capabilities including clusters and so forth
Software capabilities to meet QoS, measurements
Autonomic computing principles enable high availability of resources
VO (Virtual Organization)
Common concerns and requirements on resource sharing
Conditional, time-bound, and rules-driven resource sharing
Dynamic collection of individuals and/or institutions
Sharing relationship among participants is peer-to-peer in nature
Resource sharing based on an open and well-defined set of
interaction and access rules
Evolution of Grid Computing: Third
Generation(2000)
Service-oriented architectures
Information aspects: relationship with the World Wide Web
Live information systems
Third Generation Technologies are
Web Services
OGSA
Semantic Grid
On Demand
Autonomic
Autonomic computing
Business On Demand and Infrastructure
Virtualization
Business On Demand and Infrastructure
Virtualization
Responsive: Business On Demand has to be responsive to dynamic,
unpredictable changes in demand, supply, pricing, labor, and competition.
Variable: Business On Demand has to be flexible in adapting to variable
cost structure and processes associated with productivity, capital, and
finance.
Focused: Business On Demand has to focus on their core competency, its
differentiating tasks and assets along with closer integration with its
partners.
Resilient: A Business On Demand company has to be capable of
managing changes and competitive threats with consistent availability
and security.
Business On Demand and Infrastructure
Virtualization
Business On Demand and Infrastructure
Virtualization
Integrate: Integrated systems enable seamless linkage across the
enterprise and across its entire range of customers, partners, and suppliers.
Virtualization: Resource virtualization enables the best use of resources
and minimizes complexity for users. We can achieve the virtualization of
resources through the use of a number of existing and emerging technologies
including clusters, LPARs, server blades, and grid. Based on our earlier
discussions, we know that grids provide the best use of the virtualized
resources for its virtually organized customers within the constraints of
service-level agreements and policies.
Automation: As we discussed in the last section, the autonomic capabilities
provides a dependable technology framework for an on-demand operating
environment.
Open standards: An open, integrateable technology allows resource
sharing to be more modular. Some of the most notable open standards are
XML, Web services, and OGSA standards.
Service-Oriented Architecture and Grid
Semantic Grids
Semantic Grids
Programming Models
Grid Programming Issues
Grid Programming Models
Grid Programming Issues
Portability, interoperability, and adaptivity
Discovery
Performance
Fault tolerance
Security
Program Metamodels
Grid Programming Models
Shared-state models
Message-passing models
RPC and RMI models
Hybrid models
Peer-to-peer models
Frameworks, component models, and portals
Web service models
Coordination models
Shared State Model
Provides High Bandwidth and Low
Latency
Java Spaces
Publish / Subscribe 3 possible
ways
Network as a Server CORBA event
service
Middleware - Forwarding Layer for
Application level Peer-to-Peer Services
(FLAPPS)
Network Overlays
Message Passing Model
Message Passing Model Cont..
Processes run in disjoint address spaces, and information is
exchanged using message passing of one form or another
MPI and its variants
MPICH- G2
MagPIe Libraries - broadcast, barrier, and reduce operations with
optimizations for wide-area systems as Grids
PACX-MPI - collective operations and support for intermachine
communication using TCP and SSL
Stampi supports MPI-IO and MPI-2
MPI Connect

One Side Message Passing


MPI - 2
RPC and RMI Model
RPC and RMI cont..
Grid Enabled RPC
Dynamic resource discovery and scheduling
Security
Fault tolerance

Java RMI Truly Object Oriented


Hybrid Models
Open MP and MPI
OpenMP is a library that supports parallel programming in shared-
memory parallel machines

OmniRPC - thread-safe RPC facility for clusters and Grids


MPJ Message Passing Java
Peer to Peer Model
Sharing of computer resources and
services by direct exchange between
systems
JXTA juxtapose
set of open, generalized P2P protocols,
defined in XML messages
allows any connected device on the
network ranging from cell phones and
wireless PDAs to PCs and servers to
communicate and collaborate in a P2P
manner
self-organized and self-configured
Frameworks, component models and portals
Cactus
Cactus Code and Computational Toolkit provides a modular framework for
computational physics

CORBA
a standard tool in which a metalanguage interface is used to manage interoperability
among objects

CoG Kit
Commodity Grids

Legion
provides objects with a globally unique (and opaque) identifier

Components Architecture and Portals


Components extend the object-oriented paradigm by enabling objects to manage the
interfaces they present and discover those presented by others
Portals can be viewed as providing a Web-based interface to a distributed system
Web Service Model
OGSA - Open Grid
Services Architecture
Coordinate Model
to provide a means of integrating a number of possibly
heterogeneous components together
interfacing with each component in such a way that the
collective set forms a single application that can execute on
parallel and distributed systems
heterogeneity
Advanced Programming Support
Traditional techniques
Data-driven techniques
Speculative or optimistic techniques
Distributed techniques
Grid-aware I/O
Advanced communication services
Security
Fault tolerance
Program metamodels and Grid-aware run-time systems
Grid Security
Basic Security Comprises of
Availability
Integrity
Confidentiality

Security Concerns Deals with People and their actions.


Authentication
Authorization
Assurance
Non- Repudiation
Auditability
Grid Security Requirement
Identity
Authentication
Privacy
Integrity
Authorization
Single sign-on
Delegation
Identity & Authentication
Each entity should have an identity
Who are you?
Example: Unix login name

Authentication:
Prove your identity
Stops masquerading imposters

Examples:
Passport
Username and password
Privacy

Medical Record
Patient no: 3456
Integrity

Run myHome/rm f *

Run myHome/whoami
Message Protection
Sending message securely
Integrity
Detect whether message has been tampered

Privacy
No one other than sender and receiver should be able to read message
Authorization establishes rights to do actions
What can a particular identity do?

Examples:
Are you allowed to read this file?
Are you allowed to run a job on this machine?
Unix read/write/execute permissions

Must authenticate first


Authentication != authorization
Single sign on

Log on once
Type password once

Use any grid resource without typing password again


Delegation
Delegation

Resources on the grid can act as you


Example: Execution jobs can transfer files
Delegation can be restricted
For example: Delegation only valid for a short period of time
Some Security Terms
Cryptography
Encryption
Symmetric Key
Asymmetric Key Encryption

Decryption
Digital Certificate
Public Key Certificate
Certificate Authority
Cryptographic Keys, the building block of
cryptography, are collections of bits
The more bits that you
have, the stronger is the
key
Public key cryptography 0101001110

has two keys: 1011110111

Public key
Private key
Encryption takes data and a key, feeds it into a
function and gets encrypted data out
Encrypted data is, in <data>
principal, unreadable
unless decrypted
Encryption
Function
Symmetric Key
and
Asymmetric
Key
Decryption feeds encrypted data & a key into a
function and gets the original data
Encryption and decryption
functions are linked

Decryption
Function

<data>
Digital Signatures

Verify Aspects of the data


Who created the data
That the data has not been
tampered with
Does not stop other people
reading the data
Combine
encryption+signature
Public Key Infrastructure (PKI) provides Identity
It is set of hardware,
software, people, policies
and procedure needed to
create, manage, distribute,
use, store and revoke digital
Signature
X.509 certificate
Associates an identity with a public
key
Signed by a Certificate Authority
Certificates are similar to passports or identity
cards

State of Illinois
Name John Doe
Issuer 755 E. Woodlawn State of
Public Key Urbana IL 61801 Illinois
Seal
Validity
Signature BD 08-06-65
Male 60 200lbs
GRN Eyes
Valid Till: 01-02-2008
Certification Authorities (CAs) sign certificates
CAs are small set of trusted
entities Name
CA certificates must be Validity
Public Key
distributed securely

Issuer?
Each CA has a Certificate Policy (CP)
The Certificate Policy states:
To whom the CA will issue certificates
How the CA identifies people to whom it will issue certificates

Lenient CAs dont pose security threat because resources determine


the CAs they trust.
Security Framework by OGSI
Getting Access to Grid
Authorization and Authentication
Other Infrastructure based Security
Physical security
Operating system security
Grid and firewalls
Host intrusion detection
Network intrusion detection
What is monitor?
Monitor: use hardware and/or software tools to observes the
activities on a given system or application resource.
Analysis performance,
Detection fault,
Identify bottleneck,
Tune performance
Predict performance,
Scheduling the best resources
Where to get/put the data.
Where to execute job.
Grid Monitoring Architecture
Minimal specification to support required functionality
and allow interoperability
Scalable across wide-area network
Work with a wide range of heterogeneous resources managed
by different organizations
Minimize impact of host and network failure

Standard terminology
Share past experience / best practices
Architecture
3) Event producer &
Event schema A Directory Service
Consumer
information which supports the
publication and
discovery of producers,
4) Query 2) Lookup consumers and
or
Subscribe Directory monitoring data (events)
5) Event Service
data Producers that are the
sensors that produce
Producer 1) Event publication performance data;
information
Consumers that access
= API & wire protocol & data format and use performance
Plus security! data.
Consumer
Any program that receives monitoring data (events) from a producer can be
a consumer
Steps performed by Consumer

1. Locate events
2. Locate producers
3. Initiate a query
4. Initiate a subscription
5. Initiate an unsubscribe
6. Register
7. Accept query
8. Accept subscribe
9. Accept unsubscribe
A Directory Service
Provides information about producers or consumers that
accept requests
Main Function Supported
Authorize a search
Authorize a modification
Add
Update
Remove
Search
Producers
A software component that sends monitoring data (events) to a
consumer
Producers Steps

1. Locate event
2. Locate consumer
3. Register
4. Accept query
5. Accept subscribe
6. Accept unsubscribe
7. Initiate query
8. Initiate subscribe
9. Initiate unsubscribe
Monitoring Criteria
Scalable wide-area monitoring
Resource monitoring
Cross-API monitoring
Homogeneous data presentation
Information searching
Run-time extensibility
Filtering/fusing of data
Open and standard protocols
Security
Software availability and dependencies
Grid Monitoring Systems
Autopilot
Control and Observation in Distributed Environments (CODE)
GridICE
Grid Portals Information Repository (GPIR)
GridRM
Hawkeye
Java Agents for Monitoring and Management (JAMM)
MapCenter
Monitoring and Discovery Service (MDS3)
Mercury
Network Weather Service
visPerf
Grid Scheduling
User submits job to the
middleware
The middleware pass the
job to different scheduler
Scheduler schedules these
job to the Grid
Scheduling Paradigms
Centralized Scheduling
Distributed Scheduling
Direct Communication
Scheduling with Job Pool

Hierarchical scheduling
Centralized Scheduling
Distributed Scheduling

Via Direct Communication Via Job Pool


Hierarchical scheduling
How does Scheduling works

Resource
Resource
Discover
Selection
y

Job Schedule
Execution Generation
Resource Discovery

Pull Model Push Model


Resource Selection
Select those resources that best suit the constraints and
conditions imposed by the user, such as CPU usage, RAM
available or disk storage
Fulfills the minimum requirement
Rselected Ravailable
Schedule Generation
Divided into two parts
Job Selection
Resource Selection
First Come First Serve
Random
Priority Based
Backfilling
Job Execution
Running a single command or as complicated as running a
series of scripts that may, or may not, include set up or
staging
Grid Applications to study
Application
Federated database services
Web services
Bioinformatics
SETI@Home
Sun Grid Engine
Some National Grid Projects
GARUDA
VECC
D Grid
Thank You!!!!!!!!!!!

Вам также может понравиться