High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services

©2008 MTV Networks Japan K.K.

 Patrick Bolduan
patrick.bolduan@mtvn.jp

System Manager
MTV Networks Japan K.K.

©2008 MTV Networks Japan K.K.

 Let’s get started

©2008 MTV Networks Japan K.K.

 About MTVNJ
Company
Profile
Company Name MTV Networks Japan K.K.

President Peter Robert Bullard

Address 2-8-2 Jingumae, Shibuya-ku, Tokyo 150-0001

TEL: 03-6434-3111 FAX: 03-6434-3150

Business MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC
Description digital media services for all MTV Networks brands including MTV and
Nickelodeon in Japan. In addition to digital media services, MTVNJ
also operates a 24-hour Nickelodeon channel on CS/CATV, and the
consumer products business for Nickelodeon's character franchises in
Japan.

©2008 MTV Networks Japan K.K.

 MTVNJ Mobile Brands

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project intro

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
DB Selection

{database}

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
DB Selection

Project summary:
Merge all the database services for the MTV Flux and
MTV Mobile services on to a single database platform.

Database requirements:
• flexibility
• performance
• scalability
• redundancy
• support
• cost

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
DB Selection

MySQL MySQL
reqs/DB Oracle MS SQL
Community Enterprise

flexibility △／ × × ○ ○

performance ○ ○ ○ ○

scalability ○ ／△ ○ ○ ○

redundancy ○ ／△ ○ ○ ○

support ○ ○ × ○

cost × × ◎ △
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
DB Selection

We chose MySQL Enterprise

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
MySQL Architectures

MySQL Architectures:
• Single/multi Master
• Master/Slave
• Cluster
• DR:BD/Heartbeat HA Master

What we looked for:

• flexibility
• performance
• scalability
• redundancy
• support
• cost
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
MySQL Architectures

Single/multi Master
www www www • cheap(!)
• easy configuration
×
• load ↑ ⇒ performance ↓
• high risk (if master dies, no more DB)
Master

flexibility performance scalability redundancy support cost

○ ○/△ × × ○ ◎

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
MySQL Architectures

Master/Slave
www www www
• commonly used architecture
×
• some redundancy (read/writes separate)

• still some risk (operator needed for outage recovery)

• manual failover for outages
Master Slave

flexibility performance scalability redundancy support cost

○ △ ○ △ ○ ◎

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
MySQL Architectures

Cluster
www www www • redundant!
• can scale
△
• complicated configuration
• limited support in Japan (2007)

Cluster

flexibility performance scalability redundancy support cost

○ ○ ◎ ◎ ○/△ △

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
MySQL Architectures

DR:BD/Heartbeat HA Master
www www www • redundant (master fully redundant)
• can scale (add salves)
• reduced outage risk ◎
• can use failover for maintenance tasks
Primary Secondary
• complicated configuration

flexibility performance scalability redundancy support cost

◎ ○/△ ○/△ ◎ ○ ○

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
MySQL HA Master

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
HA Master

Application Application

VIP

heartbeat

/etc/my.cnf /etc/my.cnf
local local

/var/lib/mysql drbd /var/lib/mysql

DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
HA Master

Application Web site or whatever

VIP Manages/monitors DB process and mounts:

• provides application connection point (Virtual IP)
• monitors network status
heartbeat • watches the mysqld process
• manages the /var/lib/mysql mount point

mysqld (the database)

DRBD Synchronous device layer data replication

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
HA Master

Application 3 – heartbeat does:

Application
•4 – Failover
pDB complete!
mysql dir unmount
• DRBD 2switch
Failover
×
1 – Application
– heartbeat looses
• sDB mysqlthe
connection
detects dirtomount
problem
mysqld
VIP
process • sDB mysqld start
network or mysqld
failure
heartbeat

/etc/my.cnf /etc/my.cnf
local local

/var/lib/mysql drbd
drbd /var/lib/mysql
DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
HA Master

Application Application
3 – heartbeat does:
•14sDB
– Failover complete!
Failover mysql
3 –
– primary dir mount
heartbeat
2 – drbd
• sDBdetects
mysqld
server fails!
start
VIP outage
fails over
process
server outage

heartbeat

/etc/my.cnf /etc/my.cnf
local local

/var/lib/mysql drbd /var/lib/mysql

DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
HA MySQL install

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
HA Master Install

Old DB
architecture

A Master B Master Master

local disk local disk local disk

Dual Master Single Master

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
HA Master Install

New DB
architecture
VIP

heartbeat

/etc/my.cnf /etc/my.cnf
local local

/var/lib/mysql drbd /var/lib/mysql

DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
HA Master Install

Database migration work plan:

• stop production services
• perform full data backup (mysqldump)
• Replace MySQL architecture
• restore data from backup
• test database failover
• test application
• re-open production services

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
HA Master Install

Migration 25 –– install/configure
test
3 – application
install
1 – remove
VIP old 4 – configure VIP
process database components
drbd
connectivity
heartbeat
and mysql

heartbeat

A Master B Master Master

×
/etc/my.cnf
local disk
local ×
local disk

/var/lib/mysql drbd /var/lib/mysql

/etc/my.cnf
local disk
local

DRBD DRBD
Primary Secondary
©2008 MTV Networks Japan K.K.
 MTV Flux/MTV Mobile DB project
HA Master Install

Actual migration work times:

•production services stop – 00:10
• full data backup (mysqldump) – 00:05
• MySQL architecture replacement – 01:30
• data restore – 00:10
• database failover test – 00:15
• application test – 03:00
• production services fully restored!

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
HA MySQL lessons learned

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
Lessons learned

Impact on our application:

• essentially the same as a single master
• can use same HA for:
• single master - master/slave - multi-master
• failover fast enough that DB outages don’t affect application

Heartbeat:
• no room for configuration mistakes
• pay special attention to peer connectivity
• run lots of tests before going to production!!!

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
Lessons learned

mysqld:
• can take advantage of redundancy to swap out nodes
• never try to run both primary and secondary at once!!
• leave all the redundancy to heartbeat

DR:BD:
• requires separate partition (can be LVM)
• like heartbeat – test your configurations!!!
• always check your DRBD status after a failover
• peers sometime require a manual re-sync

©2008 MTV Networks Japan K.K.

 MTV Flux/MTV Mobile DB project
Lessons learned

Overall – good points:

• we were able to deploy HA in a short maintenance window
• commercial support available (MySQL – LINBIT)
• MySQL consultative services were a big help
• also usable with MySQL Community
• can be part of other architectures
• multi-master - master/slave

Overall – weak points:

• lots to learn, lots to test (heartbeat and DRBD)
• configuration and failover is complex
• mysqld restart requires some extra steps
• all admins/vendors need to understand failover basics
©2008 MTV Networks Japan K.K.
 HA MySQL in other MTVNJ environments

©2008 MTV Networks Japan K.K.

 Questions?

DRBD related links

http://www.drbd.org/users-guide/s-heartbeat-r1.html
http://www.drbd.org/users-guide/s-heartbeat-crm.html

MTVNJ links
http://www.mtvn.jp/en/index.html
http://www.mtvn.jp/en/service.html

Patrick Bolduan
MTV Networks Japan KK
2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001
patrick.bolduan@mtvn.jp
http://www.mtvjapan.com

©2008 MTV Networks Japan K.K.