Copyright Schmooze Com, 2013

High Availability & Disaster

Recovery Development Project
Concepts, Design and Implementation
High Availability & Disaster Recovery Development Project
CONCEPTS
Who: Schmooze Com Inc, maintainers, core developers and stewards of the FreePBX project.
WHAT: High Availability HA is the characteristic of a system to protect or recover from minor
outages in a short time frame with largely automated means.
Disaster Recovery is ensuring recoverability of your systems in the event of major system outages
whether they are caused by hardware, software, connectivity, natural, human or other means.
Each organization that runs a PBX inherently has their own pain tolerance for system downtimes. We
are in the process of developing an easily implementable, replicable, supportable HA (High Availability)
version of FreePBX for license by organizations that need a reliable HA solution.
WHERE? FreePBX is the most prevalent open source PBX platform in use in the world today, with
over 1 million active production systems, and over 20k new installations of the FreePBX Distro every
month FreePBX is used in businesses ranging from SMB to fortune 1000 companies.
WHEN? We estimate that the first private BETA will be released for testing during November 2013
WHY? We understand that your phone system is a major tool for your business, thats why keeping
your business running with minimal downtime is our goal.
Copyright Schmooze Com, 2013
BUSINESS CONTINUITY is the objective of HIGH AVAILABILITY SOLUTIONS, business continuity is
the ability to do business under any circumstance and is an essential requirement of modern
communications systems.
How Downtime Impacts Business Continuity:
Direct Cost: Cost directly attributable to the outage. device repair, shipping, external consultants, contract penalties for
delays of delivery of a service.
Additional Work Hours: not only additional IT hours to mitigate the problem, but additional hours across the company
to cover downtime.
Lost Work Hours: Sales and support calls that cant be made, 100s of office workers/call center employees cannot
work because of an outage.
Lost Revenue: Toughest to measure, but likely the most important business consequence, if sales cannot be made,
support not given on customer facing systems clients will go to competitors whose systems are working, outages can
damage your reputation and result in a loss of clients.
Regulatory Compliance and Risk Management: Certain industries and businesses must implement HA systems for
legal reasons, or to mitigate risk management within their infrastructure.
DESIGN
Development Scope of Work- Rework FreePBX platform to integrate High Availability (HA)
utilizing DRBD, Cluster Manager and Pacemaker.
Mirroring- Enable live mirroring between FreePBX nodes.
Failover- Utilizing Pacemaker, enable failover between FreePBX nodes.
Failback- When nodes recover they failback to primary FreePBX nodes.
Switchover- Enable an online management tool to switch between nodes within the
FreePBX GUI.
Managing Setup- The setup of the H/A will be done in the FreePBX GUI module and provide
simple point and click setup for your H/A system.
Monitoring and Alerting- You will be able to setup in FreePBX different levels of monitoring
and alerts that you want to receive in the event of a problem.
Documentation- Produce an administrator's guide outlining the installation and use of the
FreePBX high availability service.
Custom Development FreePBX (HA) high availability utilizing DRBD.
FreePBX will integrate DRBD based HA clustering between FreePBX deployments/nodes. The
FreePBX deployments will become separate nodes utilizing DRBD to maintain synchronization and
pacemaker monitoring to determine when system outages occur triggering the failover between nodes.
High Availability- Add the ability within FreePBX to enable node replication, clients (phones & trunks)
will talk to an active node/FreePBX deployment. The service including its IP address, can be migrated
to other nodes/deployments in that cluster at any time, either due to a failure of the active node or as an
administrative action from within the FreePBX GUI. The service will failover, or failback or automatically
by and on command by an administrator.
DRBD- DRBD can be understood as network based raid1, the bulk of the development cost will be to
rework the FreePBX platform to integrate the replication and triggers to enable failover and failback
when the system determines there is a problem with an active node.
Mirroring- DRBD works on top of block devices, i.e., hard disk partitions or LVMs logical volumes. It
mirrors each data block that it is written to disk to the peer node. Note: Data can only be accessed on
the active node. Active nodes will be mirrored to inactive nodes.
Failover- When a degraded service level is reached, this triggers a failover to a secondary (replicated
node/FrePBX deployment.) The secondary nodes IP address will be replicated to the first and phones
and trunks will reregister.
Failback- When the service is restored back to the primary server, all services are migrated back to the
primary server node.
Switchover- Using the FreePBX GUI an administrator can manually trigger a switch between FreePBX
deployments/nodes. (Useful for server maintenance windows, testing purposes or when manual control
is needed).
IMPLEMENTATION HOW?
We estimate providing several product mixes based on this development project.
Full turnkey solution, integration of all aspects of HA (SIP trunking, PBX hardware & phones and support)
Turnkey HA software, PBX appliance & phones prebuilt HA FreePBX appliances and phones.
HA software licensed FreePBX modules for configuring HA on your own HA certified hardware.
Minimum Hardware Requirements
2 x servers with minimum of 250GB of disk space
1 network interface on each machine
2GB RAM
Recommended Hardware
2 x servers with minimum of 500GB of disk space (same as above)
2 network interfaces on each machine, via different interface (eg, 1 onboard NICs, additional PCIe NIC)
8GB RAM
IPMI/DRAC/ILO functionality on servers
Configuration
Select 'Install HA Build' on install of the FreePBX Distro
All the rest is via FreePBX GUI
How is a failover triggered?
When the machine goes down or becomes unresponsive (Eg, machine B can't see machine A, machine B
will take over the roles)
When a service has failed a certain number of times (eg, asterisk unable to start, after repeated attempts)
The services we will be monitoring are Asterisk, Apache and MySQL.
Licensing
The FreePBX H/A module that will manage and setup your H/A will be a commercial paid license. Each
FreePBX server that is part of the node will need a licensed copy of the H/A module.

No Solution
Downtime in Terms of Days
Offsite Backup
Manual Restore
Downtime in Terms of Hours
Warm Spare
Manual Failover and Failback
Downtime in Terms of Minutes
High Availability
Auto Failover and Failback
Downtime in Terms of Seconds
M
i
n
u
t
e
s
H
o
u
r
s
D
a
y
s
S
e
c
o
n
d
s
CONCEPTS
Who: Schmooze Com Inc, maintainers, core developers and stewards of the FreePBX project.
WHAT: High Availability HA is the characteristic of a system to protect or recover from minor
outages in a short time frame with largely automated means.
Disaster Recovery is ensuring recoverability of your systems in the event of major system outages
whether they are caused by hardware, software, connectivity, natural, human or other means.
Each organization that runs a PBX inherently has their own pain tolerance for system downtimes. We
are in the process of developing an easily implementable, replicable, supportable HA (High Availability)
version of FreePBX for license by organizations that need a reliable HA solution.
WHERE? FreePBX is the most prevalent open source PBX platform in use in the world today, with
over 1 million active production systems, and over 20k new installations of the FreePBX Distro every
month FreePBX is used in businesses ranging from SMB to fortune 1000 companies.
WHEN? We estimate that the first private BETA will be released for testing during November 2013
WHY? We understand that your phone system is a major tool for your business, thats why keeping
your business running with minimal downtime is our goal.
High Availability & Disaster Recovery Development Project
BUSINESS CONTINUITY is the objective of HIGH AVAILABILITY SOLUTIONS, business continuity is
the ability to do business under any circumstance and is an essential requirement of modern
communications systems.
How Downtime Impacts Business Continuity:
Direct Cost: Cost directly attributable to the outage. device repair, shipping, external consultants, contract penalties for
delays of delivery of a service.
Additional Work Hours: not only additional IT hours to mitigate the problem, but additional hours across the company
to cover downtime.
Lost Work Hours: Sales and support calls that cant be made, 100s of office workers/call center employees cannot
work because of an outage.
Lost Revenue: Toughest to measure, but likely the most important business consequence, if sales cannot be made,
support not given on customer facing systems clients will go to competitors whose systems are working, outages can
damage your reputation and result in a loss of clients.
Regulatory Compliance and Risk Management: Certain industries and businesses must implement HA systems for
legal reasons, or to mitigate risk management within their infrastructure.
DESIGN
Development Scope of Work- Rework FreePBX platform to integrate High Availability (HA)
utilizing DRBD, Cluster Manager and Pacemaker.
Mirroring- Enable live mirroring between FreePBX nodes.
Failover- Utilizing Pacemaker, enable failover between FreePBX nodes.
Failback- When nodes recover they failback to primary FreePBX nodes.
Switchover- Enable an online management tool to switch between nodes within the
FreePBX GUI.
Managing Setup- The setup of the H/A will be done in the FreePBX GUI module and provide
simple point and click setup for your H/A system.
Monitoring and Alerting- You will be able to setup in FreePBX different levels of monitoring
and alerts that you want to receive in the event of a problem.
Documentation- Produce an administrator's guide outlining the installation and use of the
Copyright Schmooze Com, 2013
FreePBX high availability service.
Custom Development FreePBX (HA) high availability utilizing DRBD.
FreePBX will integrate DRBD based HA clustering between FreePBX deployments/nodes. The
FreePBX deployments will become separate nodes utilizing DRBD to maintain synchronization and
pacemaker monitoring to determine when system outages occur triggering the failover between nodes.
High Availability- Add the ability within FreePBX to enable node replication, clients (phones & trunks)
will talk to an active node/FreePBX deployment. The service including its IP address, can be migrated
to other nodes/deployments in that cluster at any time, either due to a failure of the active node or as an
administrative action from within the FreePBX GUI. The service will failover, or failback or automatically
by and on command by an administrator.
DRBD- DRBD can be understood as network based raid1, the bulk of the development cost will be to
rework the FreePBX platform to integrate the replication and triggers to enable failover and failback
when the system determines there is a problem with an active node.
Mirroring- DRBD works on top of block devices, i.e., hard disk partitions or LVMs logical volumes. It
mirrors each data block that it is written to disk to the peer node. Note: Data can only be accessed on
the active node. Active nodes will be mirrored to inactive nodes.
Failover- When a degraded service level is reached, this triggers a failover to a secondary (replicated
node/FrePBX deployment.) The secondary nodes IP address will be replicated to the first and phones
and trunks will reregister.
Failback- When the service is restored back to the primary server, all services are migrated back to the
primary server node.
Switchover- Using the FreePBX GUI an administrator can manually trigger a switch between FreePBX
deployments/nodes. (Useful for server maintenance windows, testing purposes or when manual control
is needed).
IMPLEMENTATION HOW?
We estimate providing several product mixes based on this development project.
Full turnkey solution, integration of all aspects of HA (SIP trunking, PBX hardware & phones and support)
Turnkey HA software, PBX appliance & phones prebuilt HA FreePBX appliances and phones.
HA software licensed FreePBX modules for configuring HA on your own HA certified hardware.
Minimum Hardware Requirements
2 x servers with minimum of 250GB of disk space
1 network interface on each machine
2GB RAM
Recommended Hardware
2 x servers with minimum of 500GB of disk space (same as above)
2 network interfaces on each machine, via different interface (eg, 1 onboard NICs, additional PCIe NIC)
8GB RAM
IPMI/DRAC/ILO functionality on servers
Configuration
Select 'Install HA Build' on install of the FreePBX Distro
All the rest is via FreePBX GUI
How is a failover triggered?
When the machine goes down or becomes unresponsive (Eg, machine B can't see machine A, machine B
will take over the roles)
When a service has failed a certain number of times (eg, asterisk unable to start, after repeated attempts)
The services we will be monitoring are Asterisk, Apache and MySQL.
Licensing
The FreePBX H/A module that will manage and setup your H/A will be a commercial paid license. Each
FreePBX server that is part of the node will need a licensed copy of the H/A module.

56% of enterprises in North America and 30% in Europe don't have a good disaster recovery plan, says a recent CA
Technologies survey. Of the companies surveyed, small enterprises lost, on average, more than $55,000 in revenue
due to IT failures each year, while midsize companies lost more than $91,000 and large companies lost more than
$1,000,000. A data center outage by itself can cost an average of $5,600 per minute.
CONCEPTS
Who: Schmooze Com Inc, maintainers, core developers and stewards of the FreePBX project.
WHAT: High Availability HA is the characteristic of a system to protect or recover from minor
outages in a short time frame with largely automated means.
Disaster Recovery is ensuring recoverability of your systems in the event of major system outages
whether they are caused by hardware, software, connectivity, natural, human or other means.
Each organization that runs a PBX inherently has their own pain tolerance for system downtimes. We
are in the process of developing an easily implementable, replicable, supportable HA (High Availability)
version of FreePBX for license by organizations that need a reliable HA solution.
WHERE? FreePBX is the most prevalent open source PBX platform in use in the world today, with
over 1 million active production systems, and over 20k new installations of the FreePBX Distro every
month FreePBX is used in businesses ranging from SMB to fortune 1000 companies.
WHEN? We estimate that the first private BETA will be released for testing during November 2013
WHY? We understand that your phone system is a major tool for your business, thats why keeping
your business running with minimal downtime is our goal.
BUSINESS CONTINUITY is the objective of HIGH AVAILABILITY SOLUTIONS, business continuity is
the ability to do business under any circumstance and is an essential requirement of modern
communications systems.
How Downtime Impacts Business Continuity:
Direct Cost: Cost directly attributable to the outage. device repair, shipping, external consultants, contract penalties for
delays of delivery of a service.
Additional Work Hours: not only additional IT hours to mitigate the problem, but additional hours across the company
to cover downtime.
Lost Work Hours: Sales and support calls that cant be made, 100s of office workers/call center employees cannot
work because of an outage.
Lost Revenue: Toughest to measure, but likely the most important business consequence, if sales cannot be made,
support not given on customer facing systems clients will go to competitors whose systems are working, outages can
damage your reputation and result in a loss of clients.
Regulatory Compliance and Risk Management: Certain industries and businesses must implement HA systems for
legal reasons, or to mitigate risk management within their infrastructure.
DESIGN
Development Scope of Work- Rework FreePBX platform to integrate High Availability (HA)
utilizing DRBD, Cluster Manager and Pacemaker.
Mirroring- Enable live mirroring between FreePBX nodes.
Failover- Utilizing Pacemaker, enable failover between FreePBX nodes.
Failback- When nodes recover they failback to primary FreePBX nodes.
Switchover- Enable an online management tool to switch between nodes within the
FreePBX GUI.
Managing Setup- The setup of the H/A will be done in the FreePBX GUI module and provide
simple point and click setup for your H/A system.
Monitoring and Alerting- You will be able to setup in FreePBX different levels of monitoring
and alerts that you want to receive in the event of a problem.
Documentation- Produce an administrator's guide outlining the installation and use of the
High Availability & Disaster Recovery Development Project
FreePBX high availability service.
Custom Development FreePBX (HA) high availability utilizing DRBD.
FreePBX will integrate DRBD based HA clustering between FreePBX deployments/nodes. The
FreePBX deployments will become separate nodes utilizing DRBD to maintain synchronization and
pacemaker monitoring to determine when system outages occur triggering the failover between nodes.
High Availability- Add the ability within FreePBX to enable node replication, clients (phones & trunks)
will talk to an active node/FreePBX deployment. The service including its IP address, can be migrated
to other nodes/deployments in that cluster at any time, either due to a failure of the active node or as an
administrative action from within the FreePBX GUI. The service will failover, or failback or automatically
by and on command by an administrator.
DRBD- DRBD can be understood as network based raid1, the bulk of the development cost will be to
rework the FreePBX platform to integrate the replication and triggers to enable failover and failback
when the system determines there is a problem with an active node.
Mirroring- DRBD works on top of block devices, i.e., hard disk partitions or LVMs logical volumes. It
mirrors each data block that it is written to disk to the peer node. Note: Data can only be accessed on
the active node. Active nodes will be mirrored to inactive nodes.
Failover- When a degraded service level is reached, this triggers a failover to a secondary (replicated
node/FrePBX deployment.) The secondary nodes IP address will be replicated to the first and phones
and trunks will reregister.
Failback- When the service is restored back to the primary server, all services are migrated back to the
primary server node.
Switchover- Using the FreePBX GUI an administrator can manually trigger a switch between FreePBX
deployments/nodes. (Useful for server maintenance windows, testing purposes or when manual control
is needed).
Copyright Schmooze Com, 2013
IMPLEMENTATION HOW?
We estimate providing several product mixes based on this development project.
Full turnkey solution, integration of all aspects of HA (SIP trunking, PBX hardware & phones and support)
Turnkey HA software, PBX appliance & phones prebuilt HA FreePBX appliances and phones.
HA software licensed FreePBX modules for configuring HA on your own HA certified hardware.
Minimum Hardware Requirements
2 x servers with minimum of 250GB of disk space
1 network interface on each machine
2GB RAM
Recommended Hardware
2 x servers with minimum of 500GB of disk space (same as above)
2 network interfaces on each machine, via different interface (eg, 1 onboard NICs, additional PCIe NIC)
8GB RAM
IPMI/DRAC/ILO functionality on servers
Configuration
Select 'Install HA Build' on install of the FreePBX Distro
All the rest is via FreePBX GUI
How is a failover triggered?
When the machine goes down or becomes unresponsive (Eg, machine B can't see machine A, machine B
will take over the roles)
When a service has failed a certain number of times (eg, asterisk unable to start, after repeated attempts)
The services we will be monitoring are Asterisk, Apache and MySQL.
Licensing
The FreePBX H/A module that will manage and setup your H/A will be a commercial paid license. Each
FreePBX server that is part of the node will need a licensed copy of the H/A module.

SERVICE
FILE SYSTEM
BUFFER CACHE
DISK SCHED
DISK DRIVER
RAW DEVICE
TCP/IP
NIC DRIVER
FILE SYSTEM
BUFFER CACHE
DISK SCHED
DISK DRIVER
RAW DEVICE
TCP/IP
NIC DRIVER
DISK NIC DISK NIC
SERVICE
What Pacemaker does
PACEMAKER SERVICE PACEMAKER SERVICE PACEMAKER
MACHINE IP SERVICE IP MACHINE IP
PACEMAKER SERVICE PACEMAKER SERVICE PACEMAKER
MACHINE IP SERVICE IP MACHINE IP SERVICE IP
SERVICE
F
A
I
L
O
V
E
R
F
A
I
L
B
A
C
K
CONCEPTS
Who: Schmooze Com Inc, maintainers, core developers and stewards of the FreePBX project.
WHAT: High Availability HA is the characteristic of a system to protect or recover from minor
outages in a short time frame with largely automated means.
Disaster Recovery is ensuring recoverability of your systems in the event of major system outages
whether they are caused by hardware, software, connectivity, natural, human or other means.
Each organization that runs a PBX inherently has their own pain tolerance for system downtimes. We
are in the process of developing an easily implementable, replicable, supportable HA (High Availability)
version of FreePBX for license by organizations that need a reliable HA solution.
WHERE? FreePBX is the most prevalent open source PBX platform in use in the world today, with
over 1 million active production systems, and over 20k new installations of the FreePBX Distro every
month FreePBX is used in businesses ranging from SMB to fortune 1000 companies.
WHEN? We estimate that the first private BETA will be released for testing during November 2013
WHY? We understand that your phone system is a major tool for your business, thats why keeping
your business running with minimal downtime is our goal.
BUSINESS CONTINUITY is the objective of HIGH AVAILABILITY SOLUTIONS, business continuity is
the ability to do business under any circumstance and is an essential requirement of modern
communications systems.
How Downtime Impacts Business Continuity:
Direct Cost: Cost directly attributable to the outage. device repair, shipping, external consultants, contract penalties for
delays of delivery of a service.
Additional Work Hours: not only additional IT hours to mitigate the problem, but additional hours across the company
to cover downtime.
Lost Work Hours: Sales and support calls that cant be made, 100s of office workers/call center employees cannot
work because of an outage.
Lost Revenue: Toughest to measure, but likely the most important business consequence, if sales cannot be made,
support not given on customer facing systems clients will go to competitors whose systems are working, outages can
damage your reputation and result in a loss of clients.
Regulatory Compliance and Risk Management: Certain industries and businesses must implement HA systems for
legal reasons, or to mitigate risk management within their infrastructure.
DESIGN
Development Scope of Work- Rework FreePBX platform to integrate High Availability (HA)
utilizing DRBD, Cluster Manager and Pacemaker.
Mirroring- Enable live mirroring between FreePBX nodes.
Failover- Utilizing Pacemaker, enable failover between FreePBX nodes.
Failback- When nodes recover they failback to primary FreePBX nodes.
Switchover- Enable an online management tool to switch between nodes within the
FreePBX GUI.
Managing Setup- The setup of the H/A will be done in the FreePBX GUI module and provide
simple point and click setup for your H/A system.
Monitoring and Alerting- You will be able to setup in FreePBX different levels of monitoring
and alerts that you want to receive in the event of a problem.
Documentation- Produce an administrator's guide outlining the installation and use of the
FreePBX high availability service.
Custom Development FreePBX (HA) high availability utilizing DRBD.
FreePBX will integrate DRBD based HA clustering between FreePBX deployments/nodes. The
FreePBX deployments will become separate nodes utilizing DRBD to maintain synchronization and
pacemaker monitoring to determine when system outages occur triggering the failover between nodes.
High Availability- Add the ability within FreePBX to enable node replication, clients (phones & trunks)
will talk to an active node/FreePBX deployment. The service including its IP address, can be migrated
to other nodes/deployments in that cluster at any time, either due to a failure of the active node or as an
administrative action from within the FreePBX GUI. The service will failover, or failback or automatically
by and on command by an administrator.
DRBD- DRBD can be understood as network based raid1, the bulk of the development cost will be to
rework the FreePBX platform to integrate the replication and triggers to enable failover and failback
when the system determines there is a problem with an active node.
Mirroring- DRBD works on top of block devices, i.e., hard disk partitions or LVMs logical volumes. It
mirrors each data block that it is written to disk to the peer node. Note: Data can only be accessed on
the active node. Active nodes will be mirrored to inactive nodes.
Failover- When a degraded service level is reached, this triggers a failover to a secondary (replicated
node/FrePBX deployment.) The secondary nodes IP address will be replicated to the first and phones
and trunks will reregister.
Failback- When the service is restored back to the primary server, all services are migrated back to the
primary server node.
Switchover- Using the FreePBX GUI an administrator can manually trigger a switch between FreePBX
deployments/nodes. (Useful for server maintenance windows, testing purposes or when manual control
is needed).
High Availability & Disaster Recovery Development Project
IMPLEMENTATION HOW?
We estimate providing several product mixes based on this development project.
Full turnkey solution, integration of all aspects of HA (SIP trunking, PBX hardware & phones and support)
Turnkey HA software, PBX appliance & phones prebuilt HA FreePBX appliances and phones.
HA software licensed FreePBX modules for configuring HA on your own HA certified hardware.
Minimum Hardware Requirements
2 x servers with minimum of 250GB of disk space
1 network interface on each machine
2GB RAM
Recommended Hardware
2 x servers with minimum of 500GB of disk space (same as above)
2 network interfaces on each machine, via different interface (eg, 1 onboard NICs, additional PCIe NIC)
8GB RAM
IPMI/DRAC/ILO functionality on servers
Configuration
Select 'Install HA Build' on install of the FreePBX Distro
All the rest is via FreePBX GUI
How is a failover triggered?
When the machine goes down or becomes unresponsive (Eg, machine B can't see machine A, machine B
will take over the roles)
When a service has failed a certain number of times (eg, asterisk unable to start, after repeated attempts)
The services we will be monitoring are Asterisk, Apache and MySQL.
Licensing
The FreePBX H/A module that will manage and setup your H/A will be a commercial paid license. Each
FreePBX server that is part of the node will need a licensed copy of the H/A module.

Copyright Schmooze Com, 2013
Schmooze Com Inc.
http://schmoozecom.com
(920) 886-8130
To purchase licenses of FreePBX High Availability, please log in to your
Schmooze Portal account and click on the store tab. To create your
FREE account visit http://schmoozecom.com/oss-signup.php