INCREASING DENSITY OF ASTERISK SYSTEMS

Contents
22 Executive Summary.......................................... 3 22 Introduction...................................................... 4 22 Chunk Size ...................................................... 4 22 The Sangoma Solution ..................................... 5 22 Significantly Reducing System Load .................. 6 22 Limitations ....................................................... 7 22 Conclusion ...................................................... 7 22 Appendix.......................................................... 8

02/2012

Executive Summary
Since its creation, Asterisk has been known to require a lot of CPU resources when using the default DAHDI driver to communicate with an installed VoIP-to-TDM Interface Board. This limits the ability to deploy very dense Asterisk systems, forcing implementers to move to a more costly and technically complex multi-server solution well before the capacity of the system approaches 500 simultaneous calls. However, by properly implementing Asterisk with Sangoma hardware and software, unmatched system densities can be achieved.

Increasing the Density of Asterisk Systems by 300%

Introduction
Since its creation, Asterisk has been known to require a lot of CPU resources when using the default DAHDI driver to communicate with an installed VoIP-to-TDM Interface Board. Even with the most powerful CPU, systems can often show significant CPU load related problems when running anything more than eight spans of Digital Telephony (T1/E1/J1) Boards. By using multi-core servers, this number can be pushed to sixteen spans, but there is little CPU capacity to spare at this level of density. However, by taking advantage of hardware-based echo cancellation and properly implementing a little-known feature called Adjustable Chunk Size, a single-chassis system running Asterisk can support up to forty-eight spans of Digital Telephony (1,440 simultaneous calls) without the risk of overloading the CPU.

Chunk Size
In order to make Asterisk work with TDM interfaces, audio data must be moved between the TDM Interface Boards that are installed in a system and the DAHDI API. The size of the bucket that is used to move this audio data is called the Chunk Size. The default DAHDI driver Chunk Size is eight (8) bytes. The logic behind the eight-byte default chunk size is related to the legacy analog phone network and the best practices in managing echo cancellation in software. However, once hardware-based echo cancellation and digital telephony became available, these restraints are no longer as necessary. However, the eight-byte default chuck size still remains. Only 1 millisecond of audio can be moved from the TDM Interface Hardware to the DAHDI driver in a single eight-byte bucket. Each time a bucket of data needs to be moved, a CPU Interrupt is raised. This translates into 1000 interrupts per second. An interrupt is a call for attention to the CPU. The more interrupts being generated by a system, the more load that is being generated on that CPU. From Wikipedia :

Interrupts provide low overhead and good latency at low offered load, but degrade significantly at high interrupt rate unless care is taken to prevent several pathologies. These are various forms of livelocks, when the system spends all of its time processing interrupts, to the exclusion of other required tasks. Under extreme conditions, a large number of interrupts (like very high network traffic) may completely stall the system. To avoid such problems, an operating system must schedule network interrupt handling as carefully as it schedules process execution.[1] This potentially high volume of hardware interrupts can cause Asterisk systems using DAHDI and an internal TDM interface board to suffer serious load related issues when running even a few hundred simultaneous calls.

Increasing the Density of Asterisk Systems by 300%

The Sangoma Solution

By implementing echo cancelation in specialized DSP hardware that is part of the Sangoma board, plus implementing an Adjustable Chuck Size, systems built with Sangoma hardware and software can often achieve densities of three times the size of those possible with competing configurations. These two features significantly reduce the number of interrupts to the CPU, thereby lowering the system load. Keep in mind this is just the load created by moving audio from the hardware up to the software, before anything useful has been done with that audio. Remember that any Asterisk system is more reliable under lower system load. All non-analog Sangoma hardware is able to automatically detect the DAHDI chunk size and adjust accordingly. Chunk size can be implemented at 8 bytes (1 ms default), 16 bytes (2 ms), 40 bytes (5 ms) or 80 bytes (10 ms). The parsing of the data into larger chunks reduces the number of interrupts necessary to process the same amount (milliseconds) of voice data. This reduces the interrupt loads for the driver to a small fraction of their values when compared with operation under default conditions.

Increasing the Density of Asterisk Systems by 300%

Significantly Reducing System Load

The results below are for a machine with an Intel Core 2 Quad CPU Q9550 @ 2.83GHz with 4 Gbytes of RAM. Two A108D cards provided 496 call capacity (31 channels per span x 16 E1 spans). A small user program that emulates the Asterisk DAHDI interfaces was used to exercise the driver. It avoids the variations of load in user space that we would have with Asterisk itself, and it allows us to use the channels normally used for the D-channel to be used for calls, which adds to the steady system load. Performance under real Asterisk is more variable depending on the applications being run on top.

The reductions are quite dramatic. At 496 calls, the system load is reduced by over 70% from 26% to 7% as the chunk size goes from 8 bytes (1 ms) to 80 bytes (10 ms). At idle, the reduction is from 15% down to a negligible 1%. It should be noted that the most significant impact on CPU load is created by moving from an 8-byte chunk size to a 40-byte chunk size. This is important because having a chunk size of more than 40 bytes can impact other aspects of performance when using internal software clock of Asterisk. To move beyond the 40 byte chunk size, clocking needs to also be moved to the installed Sangoma hardware.

Increasing the Density of Asterisk Systems by 300%

Limitations
1.  This solution does not work for ANALOG and T1/E1 E&M wink. This is because DAHDI currently hard-codes

the chunk size to 8 in some of the Analog/RBS timing routines. 2.  One must use a board with hardware echo cancellation. The software echo cancellation routines in DAHDI, or the OSLEC would not handle the larger chunk sizes well.

Conclusion
The DAHDI design and implementation are very robust, but DAHDI was not designed for large systems, or systems operating on marginal software. However, by properly implementing Asterisk with Sangoma hardware and software, unmatched system densities can be achieved.

Asterisk is a registered trademark of Digium Corporation.

Increasing the Density of Asterisk Systems by 300%

Appendix
CONFIGURING DAHDI FOR A LARGER CHUNK SIZE

To configure DAHDI for higher chunk size: ./Setup install --dahdi-chunk=<CHUNK SIZE> Where CHUNK SIZE is 8, 16 , 40 or 80. Setup will patch DAHDI source for the selected chunk size and will remove wct4xxx (Digium) driver from the DAHDI Makefile. The reason for the wct4xxx removal is that the wct4xxx driver does not support any chunk size other than 8 bytes (1 ms). Once Setup recompiles DAHDI and Wanpipe drivers, there are no other configuration changes needed. Just start Wanpipe, DAHDI and Asterisk: wanrouter start dahdi_cfg vvv asterisk -c -r To confirm that you are running with new dahdi chunk size, run: ifconfig w1g1 And confirm that the MTU is equal to the configured chunk size.

Increasing the Density of Asterisk Systems by 300%

MACHINE CONFIGURATION

Machine Info: cat /proc/cpu [root@tesla ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz stepping : 10 cpu MHz : 2842.866 cache size : 6144 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_ tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr lahf_lm bogomips : 5688.50 processor vendor_id cpu family model model name stepping cpu MHz cache size physical id siblings core id cpu cores fdiv_bug hlt_bug f00f_bug coma_bug fpu : : : : : : : : : : : : : : : : : 1 GenuineIntel 6 23 Intel(R) Core(TM)2 Quad CPU 10 2842.866 6144 KB 0 4 1 4 no no no no yes

Q9550

@ 2.83GHz

Increasing the Density of Asterisk Systems by 300%

fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_ tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr lahf_lm bogomips : 5685.57 processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz stepping : 10 cpu MHz : 2842.866 cache size : 6144 KB physical id : 0 siblings : 4 core id : 2 cpu cores : 4 fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_ tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr lahf_lm bogomips : 5685.59 processor vendor_id cpu family model model name stepping cpu MHz cache size physical id siblings core id cpu cores fdiv_bug hlt_bug f00f_bug coma_bug fpu : : : : : : : : : : : : : : : : : 3 GenuineIntel 6 23 Intel(R) Core(TM)2 Quad CPU 10 2842.866 6144 KB 0 4 3 4 no no no no yes

Q9550

@ 2.83GHz

Increasing the Density of Asterisk Systems by 300%

10

fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_ tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr lahf_lm bogomips : 5685.60

cat /proc/memory [root@tesla ~]# cat /proc/meminfo MemTotal: 3369408 kB MemFree: 880972 kB Buffers: 593672 kB Cached: 1377928 kB SwapCached: 0 kB Active: 910204 kB Inactive: 1285396 kB HighTotal: 2489856 kB HighFree: 868080 kB LowTotal: 879552 kB LowFree: 12892 kB SwapTotal: 923728 kB SwapFree: 923728 kB Dirty: 80 kB Writeback: 0 kB AnonPages: 223984 kB Mapped: 67616 kB Slab: 266820 kB PageTables: 5708 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 2608432 kB Committed_AS: 410352 kB VmallocTotal: 114680 kB VmallocUsed: 45960 kB VmallocChunk: 64500 kB HugePages_Total: 0 HugePages_Free: 0

Typical vmstat outputs: Chunk size= 8 bytes, no load procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 798684 591196 1459432 0 0 0 0 3027 514 0 14 86 0 0

Increasing the Density of Asterisk Systems by 300%

11

Chunk size= 8 bytes, 496 channels procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st 4 0 0 625020 589796 1459612 0 0 0 0 3071 65015 12 26 62 0 0 _______________________________________________ Chunk size= 80 bytes, no load procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 882400 593672 1377936 0 0 0 0 1226 446 0 1 99 0 0 Chunk size= 80 bytes, 496 channels procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----r b swpd free buff cache si so bi bo in cs us sy id wa st 6 0 0 706272 591500 1377936 0 0 0 0 1209 46195 12 6 81 0 0

REFERENCES
1

Jeffrey C Mogul and K.K.Ramakrishan, Eliminating receive livelock in an interrupt-driven kernel AMC Transaction on Computer  Systems, Vol 15, Issue 3 (August 1997), document ID 10.1145/263326.263335

Increasing the Density of Asterisk Systems by 300%

12

ABOUT SANGOMA TECHNOLOGIES Sangoma is a leading provider of hardware and software components that enable or enhance IP Communications Systems for both telecom and datacom applications. Enterprises, SMBs and Carriers in over 150 countries rely on Sangomas technology as part of their mission critical infrastructures. Through its worldwide network of Distribution Partners, Sangoma delivers the industrys best engineered, highest quality products, some of which carry the industrys first lifetime warranty. The product line in data and telecom boards for media and signal processing, as well as gateway appliances and software. Founded in 1984, Sangoma Technologies Corporation is publicly traded on the TSX Venture Exchange (TSX VENTURE: STC). Additional information on Sangoma can be found at http://sangoma.com

Increasing the Density of Asterisk Systems by 300%

13