11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

Documentation Documentation Home Log in

Fail Over MySQL with HAProxy

Background
Once you've established replication, you still need to ensure the applications that use this database can always find a healthy replica. This procedure takes an existing Master-Master Replication and sets up fail over between the two servers, using the excellent open source HAProxy load balancer. This lesson focuses on getting MySQL ready, and building up a simple HAProxy configuration. In production it would be wise to avoid a single point of failure by using two HAProxies, with keepalived directing traffic between them. It may also be wise to implement more detailed health checks between HAProxy and MySQL. This blog post introduces several shell scripts that can check the presence of vital databases, the status of replication, and even allow a server to voluntarily fail health check for maintenance.

There is no substitute for experience.

You replicate data because it's important. Treat your important data right, and get concrete experience with replication in our hands-on course. What's in the course?

Steps
1. Check that the Primary and Secondary Databases are Healthy Primary and Secondary should already be replicating, as at the end of the Establish Master-Master Replication procedure. SSH to the Primary: p r i m a r y . e x a m p l e . c o m Connect to MySQL and insert a new row:
p r i m a r y~$m y s q lur o o t m y s q l >i n s e r ti n t oi m p o r t a n t . s t u f fs e td e t a i l s=' R e p l i c a t i n gP r i m a r yt oS e c o n d a r y ' ; Q u e r yO K ,1r o wa f f e c t e d( 0 . 0 2s e c )

SSH to the Secondary: s e c o n d a r y . e x a m p l e . c o m Connect to MySQL and verify the row you inserted on the Primary has replicated:
s e c o n d a r y~$m y s q lur o o t m y s q l >s e l e c t*f r o mi m p o r t a n t . s t u f fo r d e rb yi dd e s cl i m i t1 ; + + + + |i d |d e t a i l s |h a p p e n e d | + + + + |2 9 6|R e p l i c a t i n gP r i m a r yt oS e c o n d a r y|2 0 1 3 0 4 1 21 4 : 0 9 : 2 6| + + + + 1r o wi ns e t( 0 . 0 0s e c ) m y s q l >

Still on the Secondary, insert another row:

m y s q l >i n s e r ti n t oi m p o r t a n t . s t u f fs e td e t a i l s=' R e p l i c a t i n gS e c o n d a r yt oP r i m a r y ' ; Q u e r yO K ,1r o wa f f e c t e d( 0 . 0 2s e c )

Returning to the Primary, verify the row has replicated:

m y s q l >s e l e c t*f r o mi m p o r t a n t . s t u f fo r d e rb yi dd e s cl i m i t1 ; + + + + |i d |d e t a i l s |h a p p e n e d | + + + + |2 9 7|R e p l i c a t i n gS e c o n d a r yt oP r i m a r y|2 0 1 3 0 4 1 21 4 : 1 0 : 2 6| + + + + 1r o wi ns e t( 0 . 0 0s e c ) m y s q l > After this step is completed:

1. On the primary, INSERT a new record in the table i m p o r t a n t . s t u f f 2. Within 1 second, that record is replicated to the secondary 3. On the secondary, INSERT a new record in the table i m p o r t a n t . s t u f f 4. Within 1 second, that record is replicated to the primary 2. Test the Application can Connect from the Proxy The database has a user configured named a p p l i c a t i o nwith the privileges a web app might require on the table i m p o r t a n t . s t u f f .
mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy 1/8

11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

SSH to the Proxy: p r o x y . e x a m p l e . c o m Make sure the a p p l i c a t i o nuser can log in from the Proxy server.
p r o x y~$m y s q lhp r i m a r y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a p W e l c o m et ot h eM y S Q Lm o n i t o r . C o m m a n d se n dw i t h;o r\ g . Y o u rM y S Q Lc o n n e c t i o ni di s2 3 S e r v e rv e r s i o n :5 . 5 . 3 1 l o gM y S Q LC o m m u n i t yS e r v e r( G P L ) C o p y r i g h t( c )2 0 0 0 ,2 0 1 3 ,O r a c l ea n d / o ri t sa f f i l i a t e s .A l lr i g h t sr e s e r v e d . O r a c l ei sar e g i s t e r e dt r a d e m a r ko fO r a c l eC o r p o r a t i o na n d / o ri t s a f f i l i a t e s .O t h e rn a m e sm a yb et r a d e m a r k so ft h e i rr e s p e c t i v e o w n e r s . T y p e' h e l p ; 'o r' \ h 'f o rh e l p .T y p e' \ c 't oc l e a rt h ec u r r e n ti n p u ts t a t e m e n t . m y s q l >e x i t p r o x y~$m y s q lhs e c o n d a r y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a p W e l c o m et ot h eM y S Q Lm o n i t o r . C o m m a n d se n dw i t h;o r\ g . Y o u rM y S Q Lc o n n e c t i o ni di s2 3 S e r v e rv e r s i o n :5 . 5 . 3 1 l o gM y S Q LC o m m u n i t yS e r v e r( G P L ) C o p y r i g h t( c )2 0 0 0 ,2 0 1 3 ,O r a c l ea n d / o ri t sa f f i l i a t e s .A l lr i g h t sr e s e r v e d . O r a c l ei sar e g i s t e r e dt r a d e m a r ko fO r a c l eC o r p o r a t i o na n d / o ri t s a f f i l i a t e s .O t h e rn a m e sm a yb et r a d e m a r k so ft h e i rr e s p e c t i v e o w n e r s . T y p e' h e l p ; 'o r' \ h 'f o rh e l p .T y p e' \ c 't oc l e a rt h ec u r r e n ti n p u ts t a t e m e n t . m y s q l >e x i t After this step is completed:

1. From the proxy, the user a p p l i c a t i o ncan connect to MySQL on the primary. 2. From the proxy, the user a p p l i c a t i o ncan connect to MySQL on the secondary. 3. Install and Configure HAProxy Install the HAProxy package on the Proxy server:
p r o x y~$s u d oy u mi n s t a l lh a p r o x y L o a d e dp l u g i n s :p r i o r i t i e s ,s e c u r i t y ,u p d a t e m o t d ,u p g r a d e h e l p e r S e t t i n gu pI n s t a l lP r o c e s s R e s o l v i n gD e p e n d e n c i e s >R u n n i n gt r a n s a c t i o nc h e c k >P a c k a g eh a p r o x y . x 8 6 _ 6 40 : 1 . 4 . 2 2 3 . 2 . a m z n 1w i l lb ei n s t a l l e d >F i n i s h e dD e p e n d e n c yR e s o l u t i o n D e p e n d e n c i e sR e s o l v e d = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = P a c k a g e A r c h V e r s i o n R e p o s i t o r y S i z e = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = I n s t a l l i n g : h a p r o x y x 8 6 _ 6 4 1 . 4 . 2 2 3 . 2 . a m z n 1 a m z n m a i n 5 1 1k T r a n s a c t i o nS u m m a r y = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = I n s t a l l 1P a c k a g e ( s ) T o t a ld o w n l o a ds i z e :5 1 1k I n s t a l l e ds i z e :1 . 4M I st h i so k[ y / N ] :y D o w n l o a d i n gP a c k a g e s : h a p r o x y 1 . 4 . 2 2 3 . 2 . a m z n 1 . x 8 6 _ 6 4 . r p m R u n n i n gr p m _ c h e c k _ d e b u g R u n n i n gT r a n s a c t i o nT e s t T r a n s a c t i o nT e s tS u c c e e d e d R u n n i n gT r a n s a c t i o n I n s t a l l i n g:h a p r o x y 1 . 4 . 2 2 3 . 2 . a m z n 1 . x 8 6 _ 6 4 V e r i f y i n g :h a p r o x y 1 . 4 . 2 2 3 . 2 . a m z n 1 . x 8 6 _ 6 4 I n s t a l l e d : h a p r o x y . x 8 6 _ 6 40 : 1 . 4 . 2 2 3 . 2 . a m z n 1 C o m p l e t e ! p r o x y~$

|5 1 1k B

0 0 : 0 0

1 / 1 1 / 1

Now edit the HAProxy config file, / e t c / h a p r o x y / h a p r o x y . c f g

p r o x y~$s u d o e d i t/ e t c / h a p r o x y / h a p r o x y . c f g

First, delete everything below the g l o b a lblock. The defaults that ship with HAProxy are tuned for web servers, not database servers.
/ e t c / h a p r o x y / h a p r o x y . c f g # #E x a m p l ec o n f i g u r a t i o nf o rap o s s i b l ew e ba p p l i c a t i o n . S e et h e #f u l lc o n f i g u r a t i o no p t i o n so n l i n e . # # h t t p : / / h a p r o x y . 1 w t . e u / d o w n l o a d / 1 . 4 / d o c / c o n f i g u r a t i o n . t x t # # -

mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy

2/8

11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

# #G l o b a ls e t t i n g s # g l o b a l #t oh a v et h e s em e s s a g e se n du pi n/ v a r / l o g / h a p r o x y . l o gy o uw i l l #n e e dt o : # #1 )c o n f i g u r es y s l o gt oa c c e p tn e t w o r kl o ge v e n t s . T h i si sd o n e # b ya d d i n gt h e' r 'o p t i o nt ot h eS Y S L O G D _ O P T I O N Si n # / e t c / s y s c o n f i g / s y s l o g # #2 )c o n f i g u r el o c a l 2e v e n t st og ot ot h e/ v a r / l o g / h a p r o x y . l o g # f i l e .Al i n el i k et h ef o l l o w i n gc a nb ea d d e dt o # / e t c / s y s c o n f i g / s y s l o g # # l o c a l 2 . * / v a r / l o g / h a p r o x y . l o g # l o g 1 2 7 . 0 . 0 . 1l o c a l 2 c h r o o t p i d f i l e m a x c o n n u s e r g r o u p d a e m o n / v a r / l i b / h a p r o x y / v a r / r u n / h a p r o x y . p i d 4 0 0 0 h a p r o x y h a p r o x y

#t u r no ns t a t su n i xs o c k e t s t a t ss o c k e t/ v a r / l i b / h a p r o x y / s t a t s . . . D e l e t ea l lo r i g i n a lc o n t e n tb e l o wt h i sp o i n t . . .

Then insert this new config at the bottom of the file:

/ e t c / h a p r o x y / h a p r o x y . c f g . . . . . . s o m ec o n t e n tn o ts h o w n g r o u p h a p r o x y d a e m o n #t u r no ns t a t su n i xs o c k e t s t a t ss o c k e t/ v a r / l i b / h a p r o x y / s t a t sm o d e7 7 7 l i s t e nM y S Q L0 . 0 . 0 . 0 : 3 3 0 6 t i m e o u tc o n n e c t1 0 s t i m e o u tc l i e n t1 m t i m e o u ts e r v e r1 m m o d et c p s e r v e rP r i m a r yp r i m a r y . e x a m p l e . c o m : 3 3 0 6 s e r v e rS e c o n d a r ys e c o n d a r y . e x a m p l e . c o m : 3 3 0 6

Restart the HAProxy service to apply the new config.

p r o x y~$s u d os e r v i c eh a p r o x yr e s t a r t S t o p p i n gh a p r o x y : S t a r t i n gh a p r o x y : After this step is completed: [ F A I L E D ] [ O K ]

1. HA Proxy is running. 2. The stats socket / v a r / l i b / h a p r o x y / s t a t shas permissions s r w x r w x r w x 3. HA Proxy has a l i s t e n e rconfigured, named M y S Q L 4. The l i s t e n e rcontains the server p r i m a r y . e x a m p l e . c o m 5. The l i s t e n e rcontains the server s e c o n d a r y . e x a m p l e . c o m 4. Test Round Robin Load Balancing through HA Proxy The HAProxy is listening on port 3306 (the default MySQL port) on the Proxy server. Connections to MySQL through the Proxy's hostname, use HAProxy to redirect to either Primary or Secondary. You can see which server HAProxy has connected you to by looking up the s e r v e r _ i d , which has to be unique on every server participating in replication. The Primary is configured with s e r v e r _ i d10. The Secondary is configured with s e r v e r _ i d20. Use the MySQL client's eflag to get the results of one command, and disconnect. Repeatedly checking the s e r v e r _ i d , shows that the HAProxy connects to a different server each time:
p r o x y~$m y s q lhp r o x y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a pe ' s h o wv a r i a b l e sl i k e" s e r v e r _ i d " ; ' + + + |V a r i a b l e _ n a m e|V a l u e| + + + |s e r v e r _ i d |1 0 | + + + p r o x y~$m y s q lhp r o x y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a pe ' s h o wv a r i a b l e sl i k e" s e r v e r _ i d " ; ' + + + |V a r i a b l e _ n a m e|V a l u e| + + + |s e r v e r _ i d |2 0 | + + + p r o x y~$

mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy

3/8

11/11/13
After this step is completed:

MySQL Documentation : Fail Over MySQL with HAProxy

1. Repeated connections to MySQL on p r o x y . e x a m p l e . c o malternate between s e r v e r _ i d10 and 20. 5. Set up a Health Check On the Primary, connect to MySQL and configure a user HAProxy can use to check the health of the two servers. This user doesn't need specific privileges, it will just need to connect and pass authentication.
m y s q l >G R A N TU S A G EO N* . *T O' h a p r o x y ' @ ' % ' ; Q u e r yO K ,0r o w sa f f e c t e d( 0 . 0 0s e c ) m y s q l >

This new MySQL user will replicate to the Secondary automatically. Test that the Proxy server can connect as the h a p r o x yuser to both the Primary and the Secondary.
p r o x y~$m y s q lhp r i m a r y . e x a m p l e . c o muh a p r o x y W e l c o m et ot h eM y S Q Lm o n i t o r . C o m m a n d se n dw i t h;o r\ g . Y o u rM y S Q Lc o n n e c t i o ni di s2 6 8 S e r v e rv e r s i o n :5 . 5 . 3 1 l o gM y S Q LC o m m u n i t yS e r v e r( G P L ) C o p y r i g h t( c )2 0 0 0 ,2 0 1 3 ,O r a c l ea n d / o ri t sa f f i l i a t e s .A l lr i g h t sr e s e r v e d . O r a c l ei sar e g i s t e r e dt r a d e m a r ko fO r a c l eC o r p o r a t i o na n d / o ri t s a f f i l i a t e s .O t h e rn a m e sm a yb et r a d e m a r k so ft h e i rr e s p e c t i v e o w n e r s . T y p e' h e l p ; 'o r' \ h 'f o rh e l p .T y p e' \ c 't oc l e a rt h ec u r r e n ti n p u ts t a t e m e n t . m y s q l >e x i t B y e p r o x y~$m y s q lhs e c o n d a r y . e x a m p l e . c o muh a p r o x y W e l c o m et ot h eM y S Q Lm o n i t o r . C o m m a n d se n dw i t h;o r\ g . Y o u rM y S Q Lc o n n e c t i o ni di s3 4 6 S e r v e rv e r s i o n :5 . 5 . 3 1 l o gM y S Q LC o m m u n i t yS e r v e r( G P L ) C o p y r i g h t( c )2 0 0 0 ,2 0 1 3 ,O r a c l ea n d / o ri t sa f f i l i a t e s .A l lr i g h t sr e s e r v e d . O r a c l ei sar e g i s t e r e dt r a d e m a r ko fO r a c l eC o r p o r a t i o na n d / o ri t s a f f i l i a t e s .O t h e rn a m e sm a yb et r a d e m a r k so ft h e i rr e s p e c t i v e o w n e r s . T y p e' h e l p ; 'o r' \ h 'f o rh e l p .T y p e' \ c 't oc l e a rt h ec u r r e n ti n p u ts t a t e m e n t . m y s q l >e x i t B y e

Now, edit / e t c / h a p r o x y / h a p r o x y . c n fto use the MySQL health check on both servers.
p r o x y~$s u d o e d i t/ e t c / h a p r o x y / h a p r o x y . c f g / e t c / h a p r o x y / h a p r o x y . c f g . . . . . . s o m ec o n t e n tn o ts h o w n l i s t e nM y S Q L0 . 0 . 0 . 0 : 3 3 0 6 t i m e o u tc o n n e c t1 0 s t i m e o u tc l i e n t1 m t i m e o u ts e r v e r1 m m o d et c p o p t i o nm y s q l c h e c ku s e rh a p r o x y s e r v e rP r i m a r yp r i m a r y . e x a m p l e . c o m : 3 3 0 6c h e c k s e r v e rS e c o n d a r ys e c o n d a r y . e x a m p l e . c o m : 3 3 0 6c h e c k

Then restart HAProxy.

p r o x y~$s u d os e r v i c eh a p r o x yr e s t a r t S t o p p i n gh a p r o x y : S t a r t i n gh a p r o x y : [ O K ] [ O K ]

Now we can ask HAProxy for the status of our servers. HAProxy established a socket at / v a r / l i b / h a p r o x y / s t a t sto take commands and show status. (It could also host a web interface.)

p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s #p x n a m e , s v n a m e , q c u r , q m a x , s c u r , s m a x , s l i m , s t o t , b i n , b o u t , d r e q , d r e s p , e r e q , e c o n , e r e s p , w r e t r , w r e d i s , s t a t u s , w e i g h t , a c t , b c k , c h k f a i l , c h k d o w n , M y S Q L , F R O N T E N D , , , 0 , 0 , 2 0 0 0 , 0 , 0 , 0 , 0 , 0 , 0 , , , , , O P E N , , , , , , , , , 1 , 1 , 0 , , , , 0 , 0 , 0 , 0 , , , , , , , , , , , 0 , 0 , 0 , , , M y S Q L , P r i m a r y , 0 , 0 , 0 , 0 , , 0 , 0 , 0 , , 0 , , 0 , 0 , 0 , 0 , U P , 1 , 1 , 0 , 0 , 0 , 4 2 , 0 , , 1 , 1 , 1 , , 0 , , 2 , 0 , , 0 , L 7 O K , 0 , 1 , , , , , , , 0 , , , , 0 , 0 , M y S Q L , S e c o n d a r y , 0 , 0 , 0 , 0 , , 0 , 0 , 0 , , 0 , , 0 , 0 , 0 , 0 , U P , 1 , 1 , 0 , 0 , 0 , 4 2 , 0 , , 1 , 1 , 2 , , 0 , , 2 , 0 , , 0 , L 7 O K , 0 , 1 , , , , , , , 0 , , , , 0 , 0 , M y S Q L , B A C K E N D , 0 , 0 , 0 , 0 , 2 0 0 0 , 0 , 0 , 0 , 0 , 0 , , 0 , 0 , 0 , 0 , U P , 2 , 2 , 0 , , 0 , 4 2 , 0 , , 1 , 1 , 0 , , 0 , , 1 , 0 , , 0 , , , , , , , , , , , , , , 0 , 0 , s h o ws t a treturns a lot of information in comma-separated

value format. Use the c u tcommand to show just the first, second, and 18th column.

p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s|c u td ,f 1 , 2 , 1 8 #p x n a m e , s v n a m e , s t a t u s M y S Q L , F R O N T E N D , O P E N M y S Q L , P r i m a r y , U P M y S Q L , S e c o n d a r y , U P M y S Q L , B A C K E N D , U P

The first column, M y S Q Lis the name of our proxy, set in the l i s t e nM y S Q Lline of our config file. The rows M y S Q L , F R O N T E N Dand M y S Q L , B A C K E N Dare generated automatically because we used the simplified l i s t e nconfiguration. More complicated configurations define the frontend and backend separately, see the HAProxy documentation.
mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy 4/8

11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

M y S Q L , P r i m a r yand M y S Q L , S e c o n d a r yare the servers, After this step is completed:

and both report that they are U P

1. HAProxy reports the Primary is U P 2. HAProxy reports the Secondary is U P 6. Configure HAProxy to be Active-Passive It is not a good idea to commit writes to both masters at the same time. The Repair Replication procedure illustrates a primary key collision, but there are many ways asynchronous replication can cause inconsistencies. Instead, as long as the Primary is available, HAProxy should always direct traffic to it. Edit / e t c / h a p r o x y / h a p r o x y . c n fto set the Secondary to be a b a c k u p .
p r o x y~$s u d o e d i t/ e t c / h a p r o x y / h a p r o x y . c f g / e t c / h a p r o x y / h a p r o x y . c f g . . . . . . s o m ec o n t e n tn o ts h o w n l i s t e nM y S Q L0 . 0 . 0 . 0 : 3 3 0 6 t i m e o u tc o n n e c t1 0 s t i m e o u tc l i e n t1 m t i m e o u ts e r v e r1 m m o d et c p o p t i o nm y s q l c h e c ku s e rh a p r o x y s e r v e rP r i m a r yp r i m a r y . e x a m p l e . c o m : 3 3 0 6c h e c k s e r v e rS e c o n d a r ys e c o n d a r y . e x a m p l e . c o m : 3 3 0 6c h e c kb a c k u p

Then restart HAProxy.

p r o x y~$s u d os e r v i c eh a p r o x yr e s t a r t S t o p p i n gh a p r o x y : S t a r t i n gh a p r o x y : [ O K ] [ O K ]

Ask HAProxy for the status of the servers again. This time, add column 20 and 21 to the c u tcommand.
p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s|c u td ,f 1 , 2 , 1 8 , 2 0 , 2 1 #p x n a m e , s v n a m e , s t a t u s , a c t , b c k M y S Q L , F R O N T E N D , O P E N , , M y S Q L , P r i m a r y , U P , 1 , 0 M y S Q L , S e c o n d a r y , U P , 0 , 1 M y S Q L , B A C K E N D , U P , 1 , 1

Adding the b a c k u poption in the config file has separated the servers into two groups, active and backup. HAProxy won't direct any traffic to any backup server until all the active servers are down. On the B A C K E N Drow, the a c tand b c kcolumns count the number of healthy servers in each state: there is one active server, and one backup server. On the server rows, the a c tand b c kcolumns tell us which type a server is. The Primary is active (1 , 0 ) and the Secondary is backup (0 , 1 ) Now, connecting repeatedly through HAProxy should always return s e r v e r _ i d10:
p r o x y~$m y s q lhp r o x y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a pe ' s h o wv a r i a b l e sl i k e" s e r v e r _ i d " ; ' + + + |V a r i a b l e _ n a m e|V a l u e| + + + |s e r v e r _ i d |1 0 | + + + p r o x y~$m y s q lhp r o x y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a pe ' s h o wv a r i a b l e sl i k e" s e r v e r _ i d " ; ' + + + |V a r i a b l e _ n a m e|V a l u e| + + + |s e r v e r _ i d |1 0 | + + + p r o x y~$ After this step is completed:

1. HAProxy shows the Primary is U Pand active. 2. HAProxy shows the Secondary is U Pand backup. 3. Repeated connections to MySQL through the Proxy always connect to the Primary 7. Test Failover by taking the Primary Down Stop the MySQL service on the Primary.
m y s q l >e x i t B y e p r i m a r y~$s u d os e r v i c em y s q l ds t o p S t o p p i n gm y s q l d :

[ O K ]

Now, on the Proxy, verify that HAProxy's health check has failed:
p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s|c u td ,f 1 , 2 , 1 8 , 2 0 , 2 1 #p x n a m e , s v n a m e , s t a t u s , a c t , b c k M y S Q L , F R O N T E N D , O P E N , , M y S Q L , P r i m a r y , D O W N , 1 , 0 M y S Q L , S e c o n d a r y , U P , 0 , 1 M y S Q L , B A C K E N D , U P , 0 , 1

mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy

5/8

11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

There are two highlighted changes here. First, the Primary has been detected as D O W N . Second, there are now 0 active servers in the M y S Q L , B A C K E N Drow, so traffic will be directed to the 1 live backup server. Connect to MySQL through the proxy:
p r o x y~$m y s q lhp r o x y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a p W e l c o m et ot h eM y S Q Lm o n i t o r . C o m m a n d se n dw i t h;o r\ g . Y o u rM y S Q Lc o n n e c t i o ni di s1 9 9 1 S e r v e rv e r s i o n :5 . 5 . 3 1 l o gM y S Q LC o m m u n i t yS e r v e r( G P L ) C o p y r i g h t( c )2 0 0 0 ,2 0 1 3 ,O r a c l ea n d / o ri t sa f f i l i a t e s .A l lr i g h t sr e s e r v e d . O r a c l ei sar e g i s t e r e dt r a d e m a r ko fO r a c l eC o r p o r a t i o na n d / o ri t s a f f i l i a t e s .O t h e rn a m e sm a yb et r a d e m a r k so ft h e i rr e s p e c t i v e o w n e r s . T y p e' h e l p ; 'o r' \ h 'f o rh e l p .T y p e' \ c 't oc l e a rt h ec u r r e n ti n p u ts t a t e m e n t . m y s q l >

Validate you've been connected to the Secondary:

m y s q l >s h o wv a r i a b l e sl i k e' s e r v e r _ i d ' ; + + + |V a r i a b l e _ n a m e|V a l u e| + + + |s e r v e r _ i d |2 0 | + + + 1r o wi ns e t( 0 . 0 0s e c )

While the Primary is down, the Secondary can keep handling read and write traffic from the application. INSERT a new record in i m p o r t a n t . s t u f f .
m y s q l >i n s e r ti n t oi m p o r t a n t . s t u f fs e td e t a i l s = ' I n s e r t e dw i t hp r i m a r yd o w n ' ; Q u e r yO K ,1r o wa f f e c t e d( 0 . 0 1s e c ) After this step is completed:

1. On the Primary, MySQL service is stopped. 2. HAProxy reports the Primary is D O W N 3. Connections to MySQL through the Proxy always connect to the Secondary 8. Restore the Primary, Verify Replication Brings it Up to Date Restart the MySQL process on the Primary:
p r i m a r y~$s u d os e r v i c em y s q l ds t a r t S t a r t i n gm y s q l d : [ O K ]

Back on the Proxy server, close your MySQL connection through the HAProxy. Verify that HAProxy detects that the Primary is back up.
m y s q l >e x i t B y e p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s|c u td ,f 1 , 2 , 1 8 , 2 0 , 2 1 #p x n a m e , s v n a m e , s t a t u s , a c t , b c k M y S Q L , F R O N T E N D , O P E N , , M y S Q L , P r i m a r y , U P , 1 , 0 M y S Q L , S e c o n d a r y , U P , 0 , 1 M y S Q L , B A C K E N D , U P , 1 , 1

Connect to MySQL through the Proxy again, and verify that you've reached the Primary:
p r o x y~$m y s q lhp r o x y . e x a m p l e . c o mua p p l i c a t i o np M o m T o o l A c e H e a p m y s q l >s h o wv a r i a b l e sl i k e' s e r v e r _ i d ' ; + + + |V a r i a b l e _ n a m e|V a l u e| + + + |s e r v e r _ i d |1 0 | + + + 1r o wi ns e t( 0 . 0 0s e c )

Check that the Primary has replicated the changes made on the Secondary while the Primary was out of service.
m y s q l >s e l e c t*f r o mi m p o r t a n t . s t u f fo r d e rb yi dd e s cl i m i t1 ; + + + + |i d |d e t a i l s |h a p p e n e d | + + + + |2 8 9|I n s e r t e dw i t hp r i m a r yd o w n|2 0 1 3 0 5 0 21 7 : 2 2 : 3 8| + + + + 1r o wi ns e t( 0 . 0 0s e c ) m y s q l >e x i t B y e After this step is completed:

1. On the Primary, MySQL service is running. 2. HAProxy reports the Primary is U P 3. Connections to MySQL through the Proxy always connect to the Primary 9. Take the Primary Offline for Maintenance It will occasionally be necessary to choose to failover to the Secondary, for example to perform maintenance or upgrades on the Primary. In that case, we'll need
mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy 6/8

11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

to tell HAProxy to ignore the health check and never send traffic to the Primary. First, we need to configure the HAProxy socket to take changes. Edit / e t c / h a p r o x y / h a p r o x y . c f g
p r o x y~$s u d o e d i t/ e t c / h a p r o x y / h a p r o x y . c f g

And update the s t a t ss o c k e tto enable admin access.

/ e t c / h a p r o x y / h a p r o x y . c f g . . . . . . s o m ec o n t e n tn o ts h o w n g r o u p h a p r o x y d a e m o n #t u r no ns t a t su n i xs o c k e t s t a t ss o c k e t/ v a r / l i b / h a p r o x y / s t a t sm o d e7 7 7l e v e la d m i n l i s t e nM y S Q L0 . 0 . 0 . 0 : 3 3 0 6 t i m e o u tc o n n e c t1 0 s t i m e o u tc l i e n t1 m . . . . . . s o m ec o n t e n tn o ts h o w n

Restart the HAProxy service to apply the new config.

p r o x y~$s u d os e r v i c eh a p r o x yr e s t a r t S t o p p i n gh a p r o x y : S t a r t i n gh a p r o x y : [ F A I L E D ] [ O K ]

Now tell HAProxy to disable the Primary server.

p r o x y~$e c h o" d i s a b l es e r v e rM y S Q L / P r i m a r y "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s|c u td ,f 1 , 2 , 1 8 , 2 0 , 2 1 #p x n a m e , s v n a m e , s t a t u s , a c t , b c k M y S Q L , F R O N T E N D , O P E N , , M y S Q L , P r i m a r y , M A I N T , 1 , 0 M y S Q L , S e c o n d a r y , U P , 0 , 1 M y S Q L , B A C K E N D , U P , 0 , 1

The Primary server reports its status as M A I N Tand does not receive any health check traffic.
After this step is completed:

1. HAProxy reports the Primary is in M A I N Tmode. 2. Connections to MySQL through the Proxy always connect to the Secondary 10. Restore Primary from Maintenance To restore service, re-enable the Primary server through HAProxy:
p r o x y~$e c h o" e n a b l es e r v e rM y S Q L / P r i m a r y "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s p r o x y~$e c h o" s h o ws t a t "|s o c a ts t d i o/ v a r / l i b / h a p r o x y / s t a t s|c u td ,f 1 , 2 , 1 8 , 2 0 , 2 1 #p x n a m e , s v n a m e , s t a t u s , a c t , b c k M y S Q L , F R O N T E N D , O P E N , , M y S Q L , P r i m a r y , U P , 1 , 0 M y S Q L , S e c o n d a r y , U P , 0 , 1 M y S Q L , B A C K E N D , U P , 1 , 1 After this step is completed:

1. HAProxy reports the Primary is U P 2. Connections to MySQL through the Proxy always connect to the Primary 11. Set HAProxy to Start Automatically By default the HAProxy service doesn't start automatically after a reboot. You can verify this by running:
p r o x y~$c h k c o n f i gl i s th a p r o x y h a p r o x y 0 : o f f 1 : o f f 2 : o f f 3 : o f f 4 : o f f 5 : o f f 6 : o f f

We want HAProxy to start automatically for any appropriate runlevel. This command starts HAProxy in all the default runlevels, excluding 0 (used to turn the system off), 1 (used to recover from severe issues), and 6 (used to reboot).
p r o x y~$s u d oc h k c o n f i gh a p r o x yo n p r o x y~$

Now check this server's current (normal) runlevel, and verify that HAProxy will start automatically the next time the server starts.
p r o x y~$r u n l e v e l N3 p r o x y~$c h k c o n f i gl i s th a p r o x y h a p r o x y 0 : o f f 1 : o f f After this step is completed:

2 : o n

3 : o n

4 : o n

5 : o n

6 : o f f

1. On the Proxy, c h k c o n f i greports that HAProxy will start automatically in runlevel 3.

"Don't take the author's word for anything; prove it to yourself. Do the exercises and invent your
mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy 7/8

11/11/13

MySQL Documentation : Fail Over MySQL with HAProxy

own." James Hague

We're happy to provide this how-to for anyone who needs it. If you're in the thick of a problem, we'd like to help where we can and stay out of your way. But if you're preparing for a project, or you'd like some experience with advanced replication features, you should check out our hands-on online course. What's in the course? Wingtip Labs 2013 Something broken or confusing? support@WingtipLabs.com

mysql.wingtiplabs.com/documentation/hap225xe/fail-over-mysql-with-haproxy

8/8