LinuxHA+DRBD+PostgreSQL

JesseYoung

TargetAudience
SystemAdministrators
DBAslookingintoreplicaBon

Goals

InspiremorereplicaBondiscussions
InformSystemAdmins/DBAs
Describecomponents
SimpleSetup
DemonstraBon
PerformanceTips

Intro
ZonarSystems,Inc.
45,000vehiclessendingdatain24/7andgrowing
15databaseservers
5TBdata,growing~25GBdaily
PostgreSQL+PostGIS

WhyuseLinuxHA+DRBD?

SimplereplicaBon
GPSdatacomingintothesystem24/7
MiBgateproblemsfromserverhardware
UsersandIntegraBonsoYwareaccess

LinuxHA
Provideahighavailability(clustering)
solu6onforLinuxwhichpromotesreliability,
availability,andserviceability(RAS)througha
communitydevelopmenteort.
SystemAdmins:YoucanconBnueto(half)
sleep,evenifaserverdies.

DRBD
DistributedReplicatedBlockDevice
Mirroredstorageacrossservers

Imagefromh_p://www.drbd.org

PostgreSQL
Databaseengineweallknowandlove

Setup

DualprocQuadcoreIntelXeon@2.66GHz
16GBRAM
2x74GBSCSI@15kRPM(RAID1OS)
6x300GBSCSI@10kRPM(RAID10DRBD)
LSIMegaraid2Channel,512MBCachew/
BBU

QuickInstall
Versions
heartbeat2.0.8
drbd8.0.5
PostgreSQLAnyversion

InstallDRBD
Compilefromsource
or
yuminstalldrbdkmoddrbd

CongureDRBD
resourcepostgres{
onnode001{
device/dev/drbd0;
disk/dev/sdb1;
address10.40.0.1:7788;
metadiskinternal;
}
onnode002{
device/dev/drbd0;
disk/dev/sdb1;
address10.40.0.2:7788;
metadiskinternal;
}
}

BeginDRBD
Eachnode:
modprobedrbd
drbdadmcreatemdpostgres
drbadmupall

CreateParBBononDRBDDevice
Primarynode:
drbdadmoverwritedataofpeerprimaryall
mke2fsj/dev/drbd0
mountdevice
Edit/etc/init.d/postgresql
PGDATAtoDRBDdevice

initdborsync/copyexisBngdatabase

DRBDstatus
cat/proc/drbd
version:8.0.5(api:86/proto:86)
SVNRevision:3011buildbyjesse@node001.zonarsystems.net,2007090223:16:51
0:cs:Connectedst:Primary/Secondaryds:UpToDate/UpToDateCr
ns:191681144nr:108784dw:191798660dr:1228217841al:962834152bm:9054lo:0
pe:0ua:0ap:0
resync:used:0/31hits:34633misses:4367starving:0dirty:0changed:4367
act_log:used:0/127hits:11433119293misses:1051777270starving:11558178
dirty:81149096changed:962834152

InstallLinuxHA
Compilefromsource
or
yuminstallheartbeat

CongureLinuxHA
ha.cf
haresources
authkeys

ha.cf
deadBme120
auto_failbacko
udpport694
ucasteth110.30.0.10
nodenode001.zonarsystems.net
nodenode002.zonarsystems.net

haresources
cluster001.zonarsystems.netIPaddr::10.30.0.101/24drbddisk::postgres\
Filesystem::/dev/drbd0::/mnt/rod1::ext3\
postgresql\
MailTo::jesse@zonarsystems.com::'Zonar_Cluster001_Switch'

authkeys
auth1
1sha1f93e5e562d430dae92b7fd7b272c13bb0c5de0e2

StartLinuxHA
/etc/init.d/heartbeatstart

DEMO

Problems?
Secondaryserveralwaysstandby
Failednodecancauseslowdowns
Kernelmodule

Performance?
ddif=/dev/zeroof=/mnt/drbd/pgwest/testbs=8192
count=2000000
DRBDdevice

~85MB/s
Diskdevice

121MB/s

30%writeperformancedegradaBon

Problems?

PerformanceImprovements

Ethernet
Directcrossovercable
GigE,10GigE
1500MTUvs9000MTU

RAIDCards
CapableofWRITEBACK
Writecaching
Ba_erybackupunit(BBU)

WritebackVS.Writethrough
Writeback
80MB/s94MB/s

Writethrough
61MB/s71MB/s

100
90
80
70
60
50
40
30
20
10
0
Run Run Run Run Run Run Run Run
1
3
5
7
9 11 13 15

DRBDProtocols
ProtocolA
DRBDwaitsforlocaldiskandlocalTCPsendbuer

ProtocolB
DRBDwaitsforlocaldiskandremotebuercache

ProtocolC
DRBDwaitsforbothlocalandremotedisk
SAFEST

YMMV
Remembertotestyourself
Fedora/e1000networkstackissues
DRBDissoYwaretoo,bugscomeup

OtherUses
HALinux+DRBD+???
Mail
NFS
MySQL

QuesBons?

Links
LinuxHA:h_p://www.linuxha.org
DRBD:h_p://www.drbd.org

Demo(backupplan)

Host:c6a

Host:c6a

Host:c6a

Host:c6a

Host:c6a

Host:c6b

Host:c6b

Host:c6a