Академический Документы
Профессиональный Документы
Культура Документы
C USTOMER S UCCESS
High Availability
MySQL Replication
Introduction 1
1 Solution 2
1.1 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Master/Slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 Master/Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.1 Audits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3.2 Managing scheduled password change . . . . . . . . . . . . . . . . . . 3
1.3.3 Backup Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Installation 4
2.1 Provided files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Preparing the Master node . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 Preparing the Slave node . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.3 Starting the replication Master/Slave . . . . . . . . . . . . . . . . . . . 8
2.2.4 Add a new Slave . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.5 Starting the replication Master/Master . . . . . . . . . . . . . . . . . . 10
2.2.6 Setting a mail notification . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 SQL replication management script . . . . . . . . . . . . . . . . . . . . . . . 11
3 Administration 13
3.1 Status of MySQL replication . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Status of the SSH tunnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Start/Stop/Restart the replication . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Backup & Restoration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4.1 Restoring a Wallix Backup . . . . . . . . . . . . . . . . . . . . . . . . 15
4 Recovery process 16
4.1 Network changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 Slave is Out of order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
©2018 WALLIX
5 Updates and Hotfix installation 18
5.1 Update the SQL replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.1.1 Checking if the replication is old . . . . . . . . . . . . . . . . . . . . . 18
5.1.2 Perform the SQL update . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2 Hotfix Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.3 Major/Minor updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.4 Update the IP addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.4.1 Update the IP address for the SQL replication . . . . . . . . . . . . . . 20
5.4.2 Update the IP address used for the SSH tunnel . . . . . . . . . . . . . 20
6 Troublshooting 21
6.1 Checking the SSH tunnel status . . . . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.1 Checking the network status . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.2 Checking the autossh service . . . . . . . . . . . . . . . . . . . . . . . 21
6.1.3 Checking the autossh configuration . . . . . . . . . . . . . . . . . . . 21
6.1.4 Check if it is possible to connect to the Master’s database from the Slave 22
6.2 Checking the SQL replication status . . . . . . . . . . . . . . . . . . . . . . . 22
6.2.1 Checking the slave status . . . . . . . . . . . . . . . . . . . . . . . . . 22
6.2.2 Checking the configuration on the Master . . . . . . . . . . . . . . . . 23
©2018 WALLIX
Introduction
The main goal of this document is to provide information regarding the High Availability so-
lution that can be implemented with WALLIX Bastions. It is based on WALLIX bash/expect
scripts and the MySQL replication mechanism within the MySQL database.
It guarantees the availability of the Bastion’s configuration on one or more devices. Hence,
working around the limitation related to the traditional HA DRBD used by the Bastion (specifi-
cally on virtual environment and devices installed on different location).
We will discuss both Master-Slave (replication on one way only) and Master-Master (replica-
tion on both ways) where the configuration (users, groups, devices, etc) and modifications will
be replicated between the Bastions.
This solution has only one limitation where only one node will be able to periodically change
passwords (see section 1.3.1).
This solution is supported by WALLIX, however it is required to contact the WALLIX pro-
fessional service for any custom modification and/or adaptation.
1 ©2018 WALLIX
C HAPTER 1
S OLUTION
1.1 Topology
1.1.1 Master/Slave
The Master/Slave mode allows the replication from one Bastion (selected as Active) to one or
several Bastions (identified as Passive). Any modification on the master will be replicated on all
the slaves. No modification should be performed from the Slaves.
1.1.2 Master/Master
The Master/Master mode allows the replication on both ways. Any modification on any Bas-
tion will be replicated on the other one.
2 ©2018 WALLIX
1.2 Principle
In order to secure the communication between the MySQL databases, a SSH tunnel with
port forwarding is established between the Bastions. This tunnel is managed by the autossh
service that keeps the tunnel permanently running. The port forwarding allows the databases to
communicate without having to open a remote access to the MySQL database.
1.3 Limitations
1.3.1 Audits
Due to its live and dynamic updates (when users are logged to targets), the audit tables are not
replicated as this can lead de database corruption. Hence, every bastion will have its own audit
session tables.
• Set the feature in the web interface of the Master in Password management > Password
change policy.
• On the slave, edit the script called for scheduled password to keep only the following line:
# vi /opt/wab/bin/WABChangeAccountCred
3 ©2018 WALLIX
C HAPTER 2
I NSTALLATION
• (Initialisation script)
install-list.sh
4 ©2018 WALLIX
2.2 Installation
2.2.1 Preparing the Master node
Attention! The following actions have to be performed as root and on the Master only
2. Check if it is possible to connect from the slave to the master on port 2242 with user
wabadmin (and vice versa).
3. Make sure that the password for wabadmin and wabsuper is the same (it can be changed
after setting up the replication).
5. Create the file servers_list on the Master and add the slave IP (one IP per line):
# cat /root/servers_list
10.10.10.1
5 ©2018 WALLIX
7. Run the script with the following argument (to do only on the Master)
# ./install-list.sh --init-master
6 ©2018 WALLIX
2.2.2 Preparing the Slave node
Attention! These steps should be performed on the Master
# ./install-list.sh --init-slave
Attention! Enter the wabadmin/wabsuper password when requested (it needs to be the same
password which can be changed after installing the replication).
Attention! After entering the passwords, the expect script will be launch and you should not
touch the keyboard during its execution.
• Connect from the Master to the Slave and configure the SSH access with the root user
using key-based authentication
7 ©2018 WALLIX
• Create the MySQL configuration for the replication
The "Current_Bastion_IP" should be the IP of the current Bastion used for establishing the
SSH tunnel on port 2242.
8 ©2018 WALLIX
• Setting the database
• Establish the SSH tunnel from the Slave to the Master (on port 3307 using autossh service)
• Set the same crypto between the nodes for database encryption
The replication is now configured in Master/Slave mode. It can be tested by creating a user
on the Master and checking its presence on the Slave.
Attention! Any modification on the slave will not be replicated on the Master and can lead to
errors.
• Create a new file /root/servers_list and add the IP of the new Slave(s)
9 ©2018 WALLIX
• Run the following command for the SSH key exchange with the Master:
# ./install-list.sh --init-master
• Create the file /root/servers_list on the Slave and add the Master’s IP
The "Current_Bastion_IP" should be the IP of the current Bastion (previously called Slave)
used for establishing the SSH tunnel on port 2242.
The replication is now working on both ways and any modification on any node is replicated
on the second one.
10 ©2018 WALLIX
2.2.6 Setting a mail notification
It is possible to set a mail notification which will allow the Slave(s) to send an email if there
is a problem with the replication.
• Run the following command as root to add execution rights for "install-list.sh":
# chmod +x install-list.sh
• Create SSH keys on master node and send them to each slave
# ./install-list.sh --init-master
• Setup SSH keys and send SQL conf file to each slave
11 ©2018 WALLIX
# ./install-list.sh --init-slave
• Copy the autossh package and setup the port forwarding from Slave to Master
# ./install-list.sh --install-tunnel Current_Bastion_IP
• Create a SQL dump send it to Slave node and setup the replication on slave
# ./install-list.sh --dump-start
• Create a SQL dump send it to Slave node and resync the slave with the Master
# ./install-list.sh --dump-resync
12 ©2018 WALLIX
C HAPTER 3
A DMINISTRATION
You need to check the following status to confirm if the replication is working:
As the synchronization is performed on the Slave (by pulling the data from the Master); it
is from the Slave that the relevant information can be retrieved (depending on which way the
replication needs to be checked).
13 ©2018 WALLIX
tcp 0 0 127.0.0.1 :3307 127.0.0.1 :XXXXX ESTABLISHED PID/ssh
• Stopping the replication can be performed by running the following command on the
Slave(s)
# mysql -uroot -padmin -e "stop slave"
• If a problem occurred during the "start" command (such as wrong log_file number), we
can use the install script from the MASTER to re-sync:
# ./install-list.sh --resync
It is possible to use the crontab and copy the scripts (with scp/rsync) to a remote storage.
14 ©2018 WALLIX
3.4.1 Restoring a Wallix Backup
If you plan to restore a full backup configuration on one of the Bastions where the replication
is running, you should consider the following:
• The replication needs to be stopped (to also restore the audit session as they are ignored
by default)
# mysql -uroot -padmin -e "stop slave"
• Double check the network configuration after restoration (especially the IP used to estab-
lish the tunnel)
15 ©2018 WALLIX
C HAPTER 4
R ECOVERY PROCESS
As the MySQL sync is performed through the SSH tunnel, it won’t be necessary to edit the
MySQL configuration.
and replace the old IP with the new one in the line: SSH_OPTIONS
If the 10 days have passed (or other non-network problems such writing issue / faulty state
occurred), a manual re-sync is required and should be performed from the Master:
• If no modification was performed on the master, the following command recovers the
replication
# ./install-list.sh --resync
• If modifications were performed on the Master (add users, targets...), you should use the
following command:
# ./install-list.sh --dump-resync
16 ©2018 WALLIX
Attention! This command overwrites the Slave’s audit with the master’s
• If modifications were performed on the Master and you want to avoid overwriting the
Slave’s audit, you can use the following command:
# ./install-list.sh --dump-resync-noauth
17 ©2018 WALLIX
C HAPTER 5
• New replication
# grep "^server-id" /etc/mysql/my.cnf /etc/mysql/conf.d/*.
cnf
/etc/mysql/conf.d/replic.cnf:server-id = X
• Old replication
# grep "^server-id" /etc/mysql/my.cnf /etc/mysql/conf.d/*.
cnf
/etc/mysql/my.cnf:server-id = X
18 ©2018 WALLIX
2. Edit the mysql configuration by commenting the following lines with #
# server-id = X
# log_bin = /var/log/mysql/mysql-bin.log
# binlog_do_db = include_database_name
# binlog_ignore_db = include_database_name
3. Stop the autossh service and delete the SSH configuration on all devices
# systemctl stop autossh
# tar cvzf /root/ssh.old.tgz /root/.ssh && rm -rf /root/.
ssh
4. Verify that the hotfix was correctly install with the command: WABVersion
19 ©2018 WALLIX
6. Check if everything is working on the Slave(s)
(if necessary, the tcp socket can be freed with the following command)
# fuser -kn tcp 3307
20 ©2018 WALLIX
C HAPTER 6
T ROUBLSHOOTING
You can follow the steps bellow to perform the initial troubleshooting steps before contacting
the Wallix support.
returning a line with the ssh PID and the ""ESTABLISHED"" status
tcp 0 0 127.0.0.1 :3307 127.0.0.1:XXXXX ESTABLISHED PID/ssh
21 ©2018 WALLIX
# cat /etc/default/autossh
# cat /etc/systemd/system/autossh.service
6.1.4 Check if it is possible to connect to the Master’s database from the Slave
The following command can be used to verify if we can reach the master’s database through
the tunnel
mysql -h127.0.0.1 -P3307 -uroot -padmin
The position can be checked by issuing the following commands (first one on the Slave and
the second one on the Master)
# mysql -uroot -padmin -e "show slave status\G" | grep
Master_Log
22 ©2018 WALLIX
# mysql -uroot -padmin -e "show master status"
23 ©2018 WALLIX