WALLIX

C USTOMER S UCCESS

High Availability
MySQL Replication

Created By Mohamed LAKHBAT

mla@wallix.com

January 4, 2019, Paris

Version 1.3 ©2018 WALLIX

Table of Contents

Introduction 1

1 Solution 2
1.1 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Master/Slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Master/Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.1 Audits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.2 Managing scheduled password change . . . . . . . . . . . . . . . . . . 3
1.3.3 Backup Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Installation 4
2.1 Provided files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Preparing the Master node . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 Preparing the Slave node . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.3 Starting the replication Master/Slave . . . . . . . . . . . . . . . . . . . 8
2.2.4 Add a new Slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.5 Starting the replication Master/Master . . . . . . . . . . . . . . . . . . 10
2.2.6 Setting a mail notification . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 SQL replication management script . . . . . . . . . . . . . . . . . . . . . . . 11

3 Administration 13
3.1 Status of MySQL replication . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Status of the SSH tunnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Start/Stop/Restart the replication . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Backup & Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4.1 Restoring a Wallix Backup . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Recovery process 16
4.1 Network changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 Slave is Out of order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

©2018 WALLIX
5 Updates and Hotfix installation 18
5.1 Update the SQL replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.1.1 Checking if the replication is old . . . . . . . . . . . . . . . . . . . . . 18
5.1.2 Perform the SQL update . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2 Hotfix Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 Major/Minor updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4 Update the IP addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.4.1 Update the IP address for the SQL replication . . . . . . . . . . . . . . 20
5.4.2 Update the IP address used for the SSH tunnel . . . . . . . . . . . . . 20

6 Troublshooting 21
6.1 Checking the SSH tunnel status . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.1 Checking the network status . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.2 Checking the autossh service . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.3 Checking the autossh configuration . . . . . . . . . . . . . . . . . . . 21
6.1.4 Check if it is possible to connect to the Master’s database from the Slave 22
6.2 Checking the SQL replication status . . . . . . . . . . . . . . . . . . . . . . . 22
6.2.1 Checking the slave status . . . . . . . . . . . . . . . . . . . . . . . . . 22
6.2.2 Checking the configuration on the Master . . . . . . . . . . . . . . . . 23

©2018 WALLIX
Introduction

The main goal of this document is to provide information regarding the High Availability so-
lution that can be implemented with WALLIX Bastions. It is based on WALLIX bash/expect
scripts and the MySQL replication mechanism within the MySQL database.
It guarantees the availability of the Bastion’s configuration on one or more devices. Hence,
working around the limitation related to the traditional HA DRBD used by the Bastion (specifi-
cally on virtual environment and devices installed on different location).

We will discuss both Master-Slave (replication on one way only) and Master-Master (replica-
tion on both ways) where the configuration (users, groups, devices, etc) and modifications will
be replicated between the Bastions.
This solution has only one limitation where only one node will be able to periodically change
passwords (see section 1.3.1).

This solution is supported by WALLIX, however it is required to contact the WALLIX pro-
fessional service for any custom modification and/or adaptation.

1 ©2018 WALLIX
 C HAPTER 1

S OLUTION

1.1 Topology
1.1.1 Master/Slave
The Master/Slave mode allows the replication from one Bastion (selected as Active) to one or
several Bastions (identified as Passive). Any modification on the master will be replicated on all
the slaves. No modification should be performed from the Slaves.

Figure 1.1 – Master/Slave

1.1.2 Master/Master
The Master/Master mode allows the replication on both ways. Any modification on any Bas-
tion will be replicated on the other one.

Figure 1.2 – Master/Master

2 ©2018 WALLIX
1.2 Principle
In order to secure the communication between the MySQL databases, a SSH tunnel with
port forwarding is established between the Bastions. This tunnel is managed by the autossh
service that keeps the tunnel permanently running. The port forwarding allows the databases to
communicate without having to open a remote access to the MySQL database.

1.3 Limitations
1.3.1 Audits
Due to its live and dynamic updates (when users are logged to targets), the audit tables are not
replicated as this can lead de database corruption. Hence, every bastion will have its own audit
session tables.

1.3.2 Managing scheduled password change

The recommended solution is to enable the scheduled password change only on the Master,
avoiding any misconfiguration between Master and Slave.
To do so:

• Set the feature in the web interface of the Master in Password management > Password
change policy.

• On the slave, edit the script called for scheduled password to keep only the following line:
# vi /opt/wab/bin/WABChangeAccountCred

Delete everything except the beginning:

#!/usr/bin/python2.7 -O

1.3.3 Backup Restoration

The replication needs to be stopped before restoring a full Bastion backup configuration (see
section 3.4.1).

3 ©2018 WALLIX
 C HAPTER 2

I NSTALLATION

2.1 Provided files

The following files are necessary to setup the SQL replication:

• (Used for login as root on the second node)

libtcl8.6_8.6.2+dfsg-2_amd64.deb
tcl-expect_5.45-6_amd64.deb
expect_5.45-6_amd64.deb
ssh.exp

• (Used to establish the ssh tunnel)

autossh_1.4d-1_amd64.deb

• (Used to pass password as argument for SSH)

sshpass_1.05-1_amd64.deb

• (Initialisation script)
install-list.sh

4 ©2018 WALLIX
2.2 Installation
2.2.1 Preparing the Master node
Attention! The following actions have to be performed as root and on the Master only

1. Copy all the provided files to the /root of the Master

2. Check if it is possible to connect from the slave to the master on port 2242 with user
wabadmin (and vice versa).

3. Make sure that the password for wabadmin and wabsuper is the same (it can be changed
after setting up the replication).

4. Install all the packages:

# dpkg -i *.deb

5. Create the file servers_list on the Master and add the slave IP (one IP per line):
# cat /root/servers_list
10.10.10.1

6. Make the script executable:

# chmod +x install-list.sh

5 ©2018 WALLIX
 7. Run the script with the following argument (to do only on the Master)

# ./install-list.sh --init-master

(enter the IP and password of the slave when asked)

The above steps perform the following:

• Generate the SSH keys (if they weren’t created previously)

• Create the MySQL configuration for the replication

• Send the SSH configuration to the Slave

6 ©2018 WALLIX
2.2.2 Preparing the Slave node
Attention! These steps should be performed on the Master
# ./install-list.sh --init-slave

Attention! Enter the wabadmin/wabsuper password when requested (it needs to be the same
password which can be changed after installing the replication).

Attention! After entering the passwords, the expect script will be launch and you should not
touch the keyboard during its execution.

The above command performs the following:

• Connect from the Master to the Slave and configure the SSH access with the root user
using key-based authentication

7 ©2018 WALLIX
 • Create the MySQL configuration for the replication

2.2.3 Starting the replication Master/Slave

Attention! These steps should be performed on the Master
# ./install-list.sh --start-all Current_Bastion_IP

The "Current_Bastion_IP" should be the IP of the current Bastion used for establishing the
SSH tunnel on port 2242.

The script requests the following confirmations:

• Installing the SSH tunnel

• Setting the encryption

8 ©2018 WALLIX
 • Setting the database

The above command performs the following:

• Copy and install the *.deb packages on the Slave

• Establish the SSH tunnel from the Slave to the Master (on port 3307 using autossh service)

• Set the same crypto between the nodes for database encryption

• Set a crontab to sync the configuration

• Dump the master database on the Slave.

• Start the replication

The replication is now configured in Master/Slave mode. It can be tested by creating a user
on the Master and checking its presence on the Slave.

Attention! Any modification on the slave will not be replicated on the Master and can lead to
errors.

2.2.4 Add a new Slave

Attention! The following procedure should be performed on the Master with root privileges.

• Rename the file /root/servers_list:

# mv /root/servers_list /root/servers_list.orig

• Create a new file /root/servers_list and add the IP of the new Slave(s)

9 ©2018 WALLIX
 • Run the following command for the SSH key exchange with the Master:
# ./install-list.sh --init-master

• Run the following command to prepare the Slave(s)

# ./install-list.sh --init-slave

• Run the following command to start the replication

# ./install-list.sh --start-all

• Merge the servers_list files

# cat /root/servers_list.orig >> /root/servers_list

2.2.5 Starting the replication Master/Master

Attention! This configuration supports only 2 Bastions
Attention! These steps should be performed on the Slave

Every Bastion is considered Master and Slave at the same time.

• Transfer the script install-list.sh to the /root of the Slave

• Create the file /root/servers_list on the Slave and add the Master’s IP

• Run the following commands:

# ./install-list.sh --install-tunnel Current_Bastion_IP
# ./install-list.sh -resync

The "Current_Bastion_IP" should be the IP of the current Bastion (previously called Slave)
used for establishing the SSH tunnel on port 2242.

The above commands perform the following:

• Establish the SSH tunnel

• Start the replication from Slave to Master

The replication is now working on both ways and any modification on any node is replicated
on the second one.

10 ©2018 WALLIX
2.2.6 Setting a mail notification
It is possible to set a mail notification which will allow the Slave(s) to send an email if there
is a problem with the replication.

Setting a mail notification manually

To do so, you will nedd to have the script WABReplicCheck.sh in /root and run the following
command:
chmod +x /root/WABReplicCheck.sh && echo "*/5 * * * * root /
root/WABReplicCheck.sh" >> /etc/cron.d/wabsynchro

Setting a mail notification using the install-list

2.3 SQL replication management script

• Run the following command as root to add execution rights for "install-list.sh":
# chmod +x install-list.sh

• Type following command to display the help section (10 options):

# ./install-list.sh

• Create SSH keys on master node and send them to each slave
# ./install-list.sh --init-master

• Setup SSH keys and send SQL conf file to each slave

11 ©2018 WALLIX
 # ./install-list.sh --init-slave

• Dump Master BDD on each slave and start replication process

# ./install-list.sh --start-all Current_Bastion_IP

• Delete all SQL log relay files on each slave

# ./install-list.sh --clean-relay

• Copy the autossh package and setup the port forwarding from Slave to Master
# ./install-list.sh --install-tunnel Current_Bastion_IP

• Share the Master ciphering key with the Slave node

# ./install-list.sh --set-crypto

• Setup the SQL replication on slave node

# ./install-list.sh --start-replication

• Create a SQL dump send it to Slave node and setup the replication on slave
# ./install-list.sh --dump-start

• Resync the slave node with the Master

# ./install-list.sh --resync

• Create a SQL dump send it to Slave node and resync the slave with the Master
# ./install-list.sh --dump-resync

• Set a mail notification

# ./install-list.sh --notification

12 ©2018 WALLIX
 C HAPTER 3

A DMINISTRATION

You need to check the following status to confirm if the replication is working:

• Status of MySQL replication

• Status of the SSH tunnel

As the synchronization is performed on the Slave (by pulling the data from the Master); it
is from the Slave that the relevant information can be retrieved (depending on which way the
replication needs to be checked).

3.1 Status of MySQL replication

To retrieve the current status, you can use the SQL command :
# mysql -uroot -padmin -e "show slave status\G" |grep "Slave_"

which should return the following output

Slave_IO_State: Waiting for master to send event
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Slave_SQL_Running_State: Slave has read all relay log; waiting
for more updates

3.2 Status of the SSH tunnel

The SSH tunnel can be verified with the following command:
# netstat -antp | grep 3307

13 ©2018 WALLIX
 tcp 0 0 127.0.0.1 :3307 127.0.0.1 :XXXXX ESTABLISHED PID/ssh

(it this case the tunnel is established)

3.3 Start/Stop/Restart the replication

The replication is managed from the Slave(s).

• Stopping the replication can be performed by running the following command on the
Slave(s)
# mysql -uroot -padmin -e "stop slave"

• Starting the replication also from the Slave(s)

# mysql -uroot -padmin -e "start slave"

• Checking the status (especially after running the start command)

# mysql -uroot -padmin -e "show slave status\G" |grep "
Slave_"

• If a problem occurred during the "start" command (such as wrong log_file number), we
can use the install script from the MASTER to re-sync:
# ./install-list.sh --resync

3.4 Backup & Restoration

No mechanism was designed on Bastion to backup the scripts. You have to follow this docu-
ment along with the scripts provided by Wallix to re-install the replication.

It is possible to use the crontab and copy the scripts (with scp/rsync) to a remote storage.

It is also recommended to use virtualisation backup mechanism such as snapshots and VM

copies to avoid date loss problems.

14 ©2018 WALLIX
3.4.1 Restoring a Wallix Backup
If you plan to restore a full backup configuration on one of the Bastions where the replication
is running, you should consider the following:

• The replication needs to be stopped (to also restore the audit session as they are ignored
by default)
# mysql -uroot -padmin -e "stop slave"

and a resync can be performed after restoration

# ./install-list.sh --resync

(to be done from every master))

• Double check the network configuration after restoration (especially the IP used to estab-
lish the tunnel)

15 ©2018 WALLIX
 C HAPTER 4

R ECOVERY PROCESS

4.1 Network changes

In case of a network change, you also need to adapt the SSH tunnel configuration.

As the MySQL sync is performed through the SSH tunnel, it won’t be necessary to edit the
MySQL configuration.

If the Master’s IP changed, edit the following file on the Slave:

# /etc/default/autossh

and replace the old IP with the new one in the line: SSH_OPTIONS

4.2 Slave is Out of order

There is a 10 days buffer on the master where data will automatically be replicated when the
faulty node comes back online.

If the 10 days have passed (or other non-network problems such writing issue / faulty state
occurred), a manual re-sync is required and should be performed from the Master:

• If no modification was performed on the master, the following command recovers the
replication
# ./install-list.sh --resync

• If modifications were performed on the Master (add users, targets...), you should use the
following command:
# ./install-list.sh --dump-resync

16 ©2018 WALLIX
 Attention! This command overwrites the Slave’s audit with the master’s

• If modifications were performed on the Master and you want to avoid overwriting the
Slave’s audit, you can use the following command:
# ./install-list.sh --dump-resync-noauth

17 ©2018 WALLIX
 C HAPTER 5

U PDATES AND H OTFIX INSTALLATION

5.1 Update the SQL replication

If the replication was deployed roughly before January 2018, it is recommended to perform
an update as its lacks consistence when performing a hotfix/minor updates.

5.1.1 Checking if the replication is old

If the following command returns a result in /etc/mysql/my.cnf, it means that the replication
is old. However, if it returns a result in /etc/mysql/conf.d then the replication is up to date.
# grep "^server-id" /etc/mysql/my.cnf /etc/mysql/conf.d/*.cnf

• New replication
# grep "^server-id" /etc/mysql/my.cnf /etc/mysql/conf.d/*.
cnf
/etc/mysql/conf.d/replic.cnf:server-id = X

• Old replication
# grep "^server-id" /etc/mysql/my.cnf /etc/mysql/conf.d/*.
cnf
/etc/mysql/my.cnf:server-id = X

5.1.2 Perform the SQL update

The following steps have be followed:
1. Stop the mysql replication (on all slaves) and the service on all devices
# mysql -uroot -padmin wallix -e "stop slave"
# systemctl stop mysql

18 ©2018 WALLIX
 2. Edit the mysql configuration by commenting the following lines with #
# server-id = X
# log_bin = /var/log/mysql/mysql-bin.log
# binlog_do_db = include_database_name
# binlog_ignore_db = include_database_name

3. Stop the autossh service and delete the SSH configuration on all devices
# systemctl stop autossh
# tar cvzf /root/ssh.old.tgz /root/.ssh && rm -rf /root/.
ssh

4. Perform the installation as explained in chapter 2

5.2 Hotfix Installation

Installing a hotfix does not require stopping the SQL replication. However, we recommend
performing the following steps:

1. Take a snapshot for each appliance

2. Install the hotfix on the Master

3. Install the hotfix on the Slave(s)

4. Verify that the hotfix was correctly install with the command: WABVersion

5. Test SSH/RDP access

5.3 Major/Minor updates

It is recommended to stop the replication before performing an update. The recommended
procedure is as follow:

1. Take a snapshot for each appliance

2. Stop the replication from the Slave(s)

# mysql -uroot -padmin -e "stop slave"

3. Update the Master as explained in the release note

4. Check if everything is working

5. Update the Slave(s)

19 ©2018 WALLIX
 6. Check if everything is working on the Slave(s)

7. Start the replication from the Slave(s)

# mysql -uroot -padmin -e "start slave"

8. Check the replication status

(force a resync if necessary using the script from the Master:)
# ./install.sh -resync

5.4 Update the IP addresses

5.4.1 Update the IP address for the SQL replication
The servers_list file can be used to update the IP(s) used by the master to reach the slave(s).

5.4.2 Update the IP address used for the SSH tunnel

Since it is always the slave who establishes the tunnel. The only IP that should be updated is
the master’s IP on the the slave’s autossh configuration.
The following procedure can be followed for that matter:

1. Stop the autossh service:

# systemctl stop autossh

(if necessary, the tcp socket can be freed with the following command)
# fuser -kn tcp 3307

2. Modify the IP address present in the file:

# vim.tiny /etc/default/autossh

3. Start the autossh service:

# systemctl start autossh

20 ©2018 WALLIX
 C HAPTER 6

T ROUBLSHOOTING

You can follow the steps bellow to perform the initial troubleshooting steps before contacting
the Wallix support.

6.1 Checking the SSH tunnel status

6.1.1 Checking the network status
The following command informs you if the tunnel is established:
# netstat -antp | grep 3307

returning a line with the ssh PID and the ""ESTABLISHED"" status
tcp 0 0 127.0.0.1 :3307 127.0.0.1:XXXXX ESTABLISHED PID/ssh

6.1.2 Checking the autossh service

The next step is to check if the autossh service (used to establish the tunnel) is running:
# systemctl status autossh

which displays the following output if the service is running:

6.1.3 Checking the autossh configuration

The parameters used to established the tunnel can be reviewed by running the following com-
mand:

21 ©2018 WALLIX
 # cat /etc/default/autossh

# cat /etc/systemd/system/autossh.service

6.1.4 Check if it is possible to connect to the Master’s database from the Slave
The following command can be used to verify if we can reach the master’s database through
the tunnel
mysql -h127.0.0.1 -P3307 -uroot -padmin

6.2 Checking the SQL replication status

6.2.1 Checking the slave status
The following command can be used to check if the Slave is synced with the Master
# mysql -uroot -padmin -e "show slave status\G" |grep "Slave_"

The position can be checked by issuing the following commands (first one on the Slave and
the second one on the Master)
# mysql -uroot -padmin -e "show slave status\G" | grep
Master_Log

22 ©2018 WALLIX
 # mysql -uroot -padmin -e "show master status"

6.2.2 Checking the configuration on the Master

The SQL replication configuration can be checked with the following command (to run on the
Master):
# cat /etc/mysql/conf.d/replic.cnf

23 ©2018 WALLIX