ProxySG SWG Initial Configuration Guide

Balancing CoreXL and SecureXL
Michael Endrizzi
Director of Services and Training
mendrizzi@midpointtech.com
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Proprietary and Confidential: No part of this document may be reproduced without permission from
1
1/24/2015 Midpoint Technology, Inc
Who Is Instructor?
• Michael Endrizzi – Midpoint Tech
• Director of Training and Services
• Age: 56
• CCSA,CCSE, CCSMA, CCISP, ITIL
• Information security since 1982
• Developer on Secure Computing
Sidewinder firewall in 1993
• Worked with NSA
• Owned information security
businesses
• Independent security consultant
for 16 years
• Working with Check Point since
1996. 10 year hiatus into auditing
now back
• Oh yeah, I like rock climbing

2
Balancing Check Point Systems
• Overview
• Linux Review
• SecureXL
• CoreXL
• Balancing CoreXL Tips

3
Danger Will Robinson
• This class will teach you how to tune
BUSY/Critical firewalls
• Commands you learn here will take
affect IMMEDIATELY
• These commands are to be used delicately.
• By issuing these commands without fully
testing them, you may negatively impact
performance on CRITICAL firewalls.
• R77.10 introduced different behavior on some
commands. Read the docs.
• Most commands do not allow you do easily
undo themselves.
• You have to use your notes and photo clips to
record the current configuration in case you
wish to revert!
• If you change multiple items at once, you may not
be able to undo the changes in case of failure.
• Backups will not save this information. You
have to do a snapshot/restore in order to recover.
• DISCLAIMER: By reading this line you are totally
responsible for all changes to your environment.

4
What is SecureXL??
SecureXL (aka Secure Network Distributor) is a way of speeding up rule processing by
analyzing traffic patterns and handing off known and previously approved traffic to
high throughput traffic handlers.
VS1
Secure Network Distributor
SND • Processing incoming traffic from the network interfaces

• Securely accelerating authorized packets (if Performance
Pack is running) - SecureXL
• Distributing non-accelerated packets among kernel
instances - CoreXL

1/24/2015 Midpoint Technology, Inc 5
What is CoreXL?
CoreXL allows you to increase the throughput capacity of your firewall
platform thru the utilization of multi-processors concurrently processing
firewall requests.
• More expensive to purchase

• Cost efficient
and maintain
• Easier to manage
• Easier to debug

6
What is NGF/VSX?
NGF (Next Generation Firewall) is a

standalone gateway running 1 instance
of the firewall module AND has the ability VSX is a physical chassis that runs multiple instances
to run several threat prevention blades of firewall gateways. Think of VMware that runs firewalls
IPS/AV/AB/Threat-Emulation as guests.
FWK VS0 VS1 VS2
Linux 2.6.18-XXcp
All based on Linux

What is SPLAT/GAIA?
• Secure Platform (SPLAT) is the commands and kernel modules added to

Linux which transforms the Linux platform into a Check Point firewall
• GAIA was a self-contained command environment created to simplify
administration. Looks like Cisco command environment.
GAIA command shell – Self

GAIA contained shell looks like Cisco
CLI
SPLAT command set added to
Linux command set
SPLAT Linux command set
Linux Kernel

Goal
NGF/VSX + CoreXL + SecureXL = Tuned System
Describes many topics in these articles. After this

course you will be able to understand these SK’s

9
Course Take Away Slide
This is the take-away slide, the value behind the whole course. If I bore you to
death with 200+ slides, this slide #190(approx) puts to practice what the whole
course is trying to teach:
Allocate CPUs in the following priority order:
1. Share cache for common data

2. Allocate CPUs to busy Internal interfaces
3. Allocate CPUs to slower less busy External interfaces
4. Allocate CPUs to FW instances
5. Remaining threads are usually idle so distribute evenly
and let kernel find idle processor

10
What Do I Need To Know?
Basic Linux CLI experience, What is operating system, What are processes, What is a
processor, What is a cache, . Basic programming: code, data, variables.
Unix ‘top’
Unix ps
/proc file system

11
Secure/CoreXL Tuning Areas
5 areas of tuning
3) CP Process
VS1 VS2
2) Fw kernel 4) Linux
Affinity Instance
Process
Affinity
FWD syslogd
logging
1) Interface
Affinity SecureXL 0) Rule Processing
SND Secure Speedup
Network
Distributor

12
Firewall Management
SmartCenter Components
Smart Dashboard
POLICY
User Space Edit Policy
CPD Push Policy FWM

FWD (mgt server (Management
(logging) communication) Server)
Kernel Space
Linux TCP/IP
NIC NIC
13
1/24/2015 Proprietary and Confidential: No part of this document may be reproduced without permission from
Midpoint Technology, Inc
NGF Components
Smart Center MGT Station

POLICY
User Space FWSSD
FWD (spawns security
(logging) servers like SMTP
filtering)
CPD VPND
CPWD
(mgt server (VPN)
(watchdog for
communication) dead processes)
Kernel Space
SPLAT/GAIA Kernel (fwk)
(Security Enforcement)

14
Smart
Center MGT
VSX/VS Components
Station
cpd cpd cpd
fwk fwk fwk
fwd fwd fwd
vpnd vpnd vpnd
User mode VS0 VS1 VS2

Kernel mode

15
R75.40 VSX – FWK in User Mode
cpd
• Note: fwk was moved to user mode
fwk
• With large number of VS’s, kernel was
getting too big
fwd
vpnd
User mode VS0

Our Lab
Eth0: 10.2.1.253/24 Eth1: 172.17.1.2/24
Eth0: 10.2.1.101/24
Eth0 : 172.17.1.111/24
Eth3: 172.17.2.2/24
Eth0:1: 10.2.2.101/24 Eth2: 10.2.2.253/24
Eth0:2 : 172.17.2.111/24
Eth0:2: 10.2.0.101/24 Eth0:1 : 172.17.0.111/24
Eth0: 10.2.1.153/24
Eth1:1 : 172.17.0.1/24
VB: internal/internet
Eth0:1 : 10.2.0.153/24
VB: Host/Host#1
20
1/24/2015
20
• Overview
• Linux Review
• SecureXL
• CoreXL

21
Linux Kernel Basics
• Linux Overview
• Threads
• Network Processing

22
Linux History

23
Evolution of the Kernel
Computer Day 1 Multics-Unix NT - Mach Linux
The User User User
Code
Blob
User User User
User User User
Monolithic Kernel
Kernel Mod Mod
Mod ule
Kernel ule ule
Kernel
• Some tried to simplify the

kernel…. • BACK to monolithic kernel (one
• Programs are • Users have their own process • Kernel broke self into smaller memory space). Minimal context
one compiled resources (memory, files..) processes, some run in user switch and no slow IPC
binary all • Monolithic kernel has its own space • Dynamic added/subtracted modules.
superuser resources (memory, code • Processes used Inter Process • User processes don’t call kernel
• PROBLEM: base). Communication to work process, user processes go into
hacking, bad • Kernel could multiprocess together kernel mode with shared data
code user processes but not self structures! Minimize context switch
corrupted • PROBLEM: SLOW In-efficient and data passing (more later on this)
whole system PROBLEM: too big expensive because of context switches
for hardware at time Proprietary and Confidential: No part of this document may be reproduced without permission from
24
Linux Differentiators
• Original kernels were monolithic:

• Single binary process
• Single processing thread
• Cooperative Multi-tasking – Could not pre-empt kernel processing
• Single Address space
• Linux
• Linux Tervald – Still heavily involved vs design by committee vs free-for-all
• Pre-emptive kernel – Most kernel tasks can be pre-empted for higher
priority tasks
• Modular – Kernel functions can be dynamically created/removed
Check Point implements firewall subsystem in these modules
• Multi-processor support
• Threads = Processes (Unique to Linux – will explain later)
• Users can see internal kernel data in sysfs file system. Looking glass
into kernel internals

25
Linux 2.4 to 2.6
Introduced to reduce the time a kernel task held on to the processor locking
out other tasks. Overall increase in efficiency and multi-processing support.
1. Scheduler – Improved fair scheduling with 100’s of processors

2. Threads – Process = thread. No special handling for threads
3. Interrupts – Can be pre-empted and no locking out all CPUs while
processing interrupts
4. Pre-emptive Kernel – The whole kernel is pre-emptive. Can be
interrupted at any point

26
Linux Kernel Structure
An application process consists of:
• User Identity & Permissions
• Code and Execution Pointer When applications call the kernel,
• Data Scratch Pad (Stack and the kernel does not “take over” as a
Hash) separate entity. Applications
• Check Point VSX Firewall Kernel transform into the “The Hulk”. The
and Helpers application processing thread takes
on kernel permissions, resources
and code base to complete a task. If
the thread does not go to sleep,
processing returns to application
mode (Bruce Banner) almost as if it
made an internal function call.
Most kernel work done

here. Talks to hardware, handles
• Resource allocation hardware interrupts.
• Scheduling In Check Point, drivers are
• Security right from manufacturer.
• Memory Mgt Some appliances have
• File System Mgt modifications
• Communication
• Check Point NGF
FirewallKernel
http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468
27
Linux Kernel Basics
• Linux Overview
• Threads

28
What is a program?
Simple terms: A program is made up of:
1. CPU Instructions – Does the work
2. Data – Scratch pad area
3. Security attributes - restrictions
• CPU Instructions
• Data
• Security attributes

29
What is a process?
A program is a compiled binary sitting on a disk in a file.
A process is when the program is executed on a processor, assigned memory, and
is managed by the OS kernel.
Program becomes a Process

• Data

30
What Does 32-bit User Program
Data look like?
All 32-bit programs have 4 gig
of memory available to them.
This is how it is allocated.

31
Why Care?:
CPU vs. Memory problem
Are you chasing a CPU or memory problem? Need to know how lack of
memory will slow a system and make it feel like it is a CPU problem.
‘top’->’f’->’u’

VM Why do you care?
If you ever want to figure out why a process is swapping, you need
to be able to know what parts of a process are taking too much space.
Virtual
Size
Physical
Memory

33
Stack Frame
A program is broken into functions. When a function
is called, the program has to save information about
where it was and pass information to the new function.
This placeholder is called a stack frame. Stack frames are To High Memory Stack
like a track of cookie crumbs to help you go back to where
External Environment
where.
Parameters
GROW
Program Counter
Saved Frame Pointer
Local Variables
To 0 memory

35
Heap space and Memory Leaks
Programs sometimes need dynamically
allocated and sized chunks of memory for maybe network
packets. The program uses the malloc call to get that
memory. It came off the heap space.
If the programmer forgets to free the memory, then you have

a memory leak.
Sometimes heap space becomes so fragmented with mixed

free and allocated memory, that programs slow down
because its hard for them to claim and release memory
efficiently. Thus ‘reboot!!!’.

36
Buffer Overflow
Stack Code
Length: 1000 bytes
Length: 256 bytes
301 LOGIN:
+ username +
Packets
Mail Fragmented
Program Running as Super-User

IMAPD Attack
Authentication Attempt Backfires
To 0 memory Stack Attack
Login
Local Variables String
Mike
UserName[256] Password
Saved Frame Pointer
Program Counter New PC

EndOfBuffer
Parameters
NULL
Environment
To High Memory

38
Virtual Memory
4GB Process A 4GB Process B 4GB Process C 4GB Process D
NOTE: Most programs don’t use all 4 gig at

once. Only use small portion.
How much total memory is needed
to store all these processes - MAX???
NOTE: This is for 32-bit systems. 64-bit can
see 264 = 16 exibytes of memory
Not needed right now
+
Virtual memory is 3 things: so swap out
1) Allows processes to think they
own all memory
10GB Disk Swap space
2) Allows processes to ignore
physical memory limitations Assuming all processes needed 3GB User space
We would need at a minimum
3) Paging system: That uses disk 12GB (3GB+3GB+3GB+3GB)User + 1GB kernel = 13GB
to swap out sleeping data to Physical and Swap space to hold all these programs.
temp storage area to free up
physical memory for active
processes.

Demo: Using VIRT Memory
Here in my ‘memgrow’ program I allocated 1GB of HEAP memory, but didn’t

read/write to it. Notice how the kernel allocated the VM space to my process but
did not actually map it into physical memory or SWAP it out because we are
running out of space (I only have 1 gig of physical memory)

DEMO: Writing to VIRT Memory
Here the same ‘memgrow’ program BUT I wrote to every byte so the kernel had
to map the page to my process and bring it into physical memory. And when physical
memory fills up…then swap it out…
WHAT IS VIRTUAL MEMORY????
1) Process thinks it owns all 32-bit 3GB by itself
2) Process doesn’t understand physical memory constraints of 1GB (only 32/64
max addressable memory
3) Paging swapped out data that the kernel could not keep in physical memory

User Program Security Attributes
Computers Day 1 allowed a program to see and do everything. But programs
started to overrun each other. So Day 2 a kernel was developed that was the
traffic cop between programs so they wouldn’t bump into each other OR
corrupt the kernel itself.
NOTE: On many embedded systems there is no kernel/user/security separation.

All processes can see and do everything. For example: SCADA controllers that run
machinery. Programs that monitor the brakes in your car.
User Level
• CPU Instructions • CPU Instructions
• Data • Data
• Security attributes • Security attributes
Kernel Level

43
What is the Kernel?
The ‘kernel’ is responsible for resource management.
User Level
• Access to physical devices

• Security separation
• Arbitrator of shared resources – who gets what first
• Virtual memory management
• Clock ticks and time slices
• Process management
• Scheduling processes
• File system access
• Network routing
Kernel Level

User Program Entering The Kernel
Linux TODAY: When a process makes a system call, the process does not hand over processing to
another entity. The processing thread gains access to kernel resources
(after appropriate security checks) so the kernel is actually executing “on behalf
of a specific process”. The process running in kernel mode now has access to both
user and kernel memory. Its like a program making a function call and
the function has enhanced magic powers, and then returns to the main user process.
Just like “The Hulk”
User Space
Kernel Space SYSTEM CALL

(security checks)
+
• Kernel Data Structures
• Superuser privileges
My friend Arah becoming the HULK
© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com 45

3 Processing Contexts
In Linux a processor can be in 1 of 3 processing contexts at any time….period
1. In user-space, executing user code in a user

process
2. In kernel-space, in process context,
executing on behalf of a specific user
process
3. In kernel-space, in interrupt context, not
associated with a process, handling an
interrupt
Why do you care?
http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/dp/0672329468

46
Pre-Emption and Context Switch
In order to avoid one process from dominating a processor pre-emptive multi-tasking was
introduced. This is where one process B can interrupt another process A at any time (pre-
empt) resulting in a context switch to the process B.
Prior to mid-1990’s context switches (stopping one process and starting another)
were CPU/Memory intensive. Kernels had to copy the process state into internal kernel
Process Control Blocks (PCBs) to save the state and then copy out a PCB to the
new process.
Process A Process B
• Virtual Memory Space

• Stack and Hash Tables Context Switch • Virtual Memory Space
(Scratch Pad) • Stack and Hash Tables
• CPU Instructions (Scratch Pad)
• Processor registers • CPU Instructions
• Security attributes • Processor registers
Kernel
Process Control Block Process Control Block

47
Old Interprocess Communication
Copy Data
In the old days communication between processes and to/from the kernel data
was copied. Very inefficient.
Process A Process B
Copy this to Process B
Kernel

48
New Interprocess Communication
Share Data
So a shared memory mechanism was developed using virtual memory tricks
that allowed memory spaces to be shared between processes and the kernel.
Process A Process B
You can see some of my data
Kernel

49
What is a Multi-Processor
Multi-processors come in several different varieties
• Multi-Processor: Multiple physical CPUs in a single chassis
• Multi-Core : Multiple processor cores on a single physical CPU
• HyperThreading: Simulated multiple processors with threads on a single physical core
• Combination of the above
• No matter which of the above configurations, the OS sees logical processors. The implementation is opaque.
• Note where the caches are! Remember this when you assign interfaces so you keep the cache hot
OS sees 4
‘logical’
processors
Logical
Processors
Threads
Cores
Physical
CPU http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx
50
What are CPU Threads
Enabling Symmetric Multi-Threading (SMT) or HyperThreading (HT) doubles the number of logical processors.
• Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting
pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel
processing.
• Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time.
• With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently.
So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time
slices.
• Performance improvement 30%??? on CPU intensive items. I/O intensive theoretically could slower.
• Shares a cache
Without Hyperthreading, Managed by Kernel

Thread 1 Thread 2 Thread 1 Thread 2
WithHyperthreading, Managed by HW
TIME
http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx

51
Example:HP-DL380p Multi-Core
No HyperThreading
HP DL380p –
CACHE
2 Physical Processors
8 Logical Processors
CPU0
CPU1
CACHE
OS sees 8
HP DL380p – 8 Logical Processors ‘logical’
processors
No Hyper
Logical
Threading Processors
Threads
Cache
Cores
Physical
CPU
52
Example: CheckPoint 12600
12 core

53
What is a multi-processing?
Day 3 – As loads increased computer designers decided to add multiple processors to the system.
Processes lent themselves nicely to the paradigm, each process could float to a free processor and
execute.
Process A Process B

• Stack and Hash Tables • Virtual Memory Space
(Scratch Pad) • Stack and Hash Tables
• CPU Instructions (Scratch Pad)
• Processor registers • CPU Instructions
• Security attributes • Processor registers

54
Shared Data Contention
LOCKUP!!
The biggest problem with concurrency is access to shared data. If not programmed
correctly two processes can fight over updating a shared piece of data and they
fight to the death. What you see is a frozen monitor!!!
• Virtual Memory Space • Virtual Memory Space

• Stack and Hash Tables • Stack and Hash Tables
(Scratch Pad) (Scratch Pad)
• CPU Instructions • CPU Instructions
• Processor registers • Processor registers
• Security attributes • Security attributes NO! I’m going to write to
I’m going to write to variable X first!!!
variable X first!!!

Threads To The Rescue
If a process wanted to enable concurrent processing (web server serving up pages, word
processor with multiple open documents), running multiple processes was inefficient.
Threads were created to support concurrent processing using SHARED data and NOT copying
data between processes or resource heavy context switching.
SHARED DATA
• Processor state
• Kernel threads
• File system
Thread C
• Signals
Thread B
Thread A

56
Light Weight/Quick Context Switches
Thread context switches are very fast because the kernel only has to save a several
registers (TCB) (approx 96 bytes vs. approx 1K+ with heavy process switches). This is
because all the non-saved data is shared between the threads so its live data and no
need to save it and restore it. In addition, threads can see into each others address space
because remember everything in the process is shared.
Thread A Thread B
• Thread registers Mini- Context Switch
• Program counter
• Stack pointer
Kernel
Thread Control Block(TCB) Thread Control Block(TCB)

https://courses.cs.washington.edu/courses/cse451/11sp/section/kim_section4.pdf
57
Threads and Multi-Processing
Threads naturally lend themselves to multi-processing…you can concurrently
run several threads on different processors.
Of course there are shared data contentions that must be dealt internally by
the threads
SHARED DATA
• Processor state
• Kernel threads
• File system
Thread C
• Signals
Thread B
Thread A

58
Threads and Scheduling
Linux 2.6.18 changed its default scheduling to favor low latency processing for more real-
time applications. It is called Completely Fair Scheduling (CSF). Its simple, when the kernel
schedules a process it chooses a process that has used the least amount of its time slice.
That way CPU-lite processes are favored. These are usually ones that are blocking waiting
for input like network interrupts, keyboard, graphics, etc.
When a process is ready to run, the kernel looks for:
1. What CPU is free on the affinity list for the process.
2. Did it run on the CPU before so the cache is fresh with my data?
3. Am I still in default mode where no autobalance occurred and so choose CPU 0
(mostly interrupts)
Thread B Thread C
Which CPU?
Thread A
Which CPU?

59
Processor Affinity
If you have a thread that is usually very busy, you can wire it to a single CPU (or subset of CPUs).
That way all the data is cached locally on the single CPU and it can usually run less interrupted on that
CPU. This is called Processor Affinity. A process/thread has an ‘affinity’ for a CPU. Can be done in Unix
and Windows Oss Check Point leverages this in their CoreXL technology.

• Processor state
• Kernel threads
• File system Thread C
• Signals
Thread B
Thread A
CPU1 CPU2 CPU3

60
Types of Threads
3 types of threads:
1. User threads: Totally internal to process, kernel cannot see them limited multi-processing
capabilities for example they couldn’t block to sleep or I/O.
2. User threads mapped onto kernel threads. Full multi-processing capabilities for user process
3. Internal kernel threads: Used only by kernel for internal kernel processing not visible
to user processes
http://linuxgazette.net/23/flower/threads.html
http://www.thegeekstuff.com/2012/03/linux-threads-intro/
User Level Internal kernel threads

• Processor state
• Kernel threads
Kernel Level User Threads mapped to Kernel Threads
Internal Kernel Threads

• Scheduler
• Blocking I/O handling
• Thread table
• Monitoring
61
Thread Demo
Internal kernel threads have a [] around them. User threads, usually do NOT have [] on
them. NOTE: sometimes user processes do, but its because the kernel can’t find its
command parameters and notates it with [] which is confusing. Other commands can
filter this out.
“ps –ef”

62
Thread Demo: Windows Threads
Even windows has threads

63
Demo: Kernel Thread
Kernel threads have no virtual memory size because they all share
the VM of the kernel and no allocated extra memory like user threads can
be allocated.
Remember this: will be similar when we look at CP NGX

64
Thread Demo: Multi-Processing Threads
You can watch threads switching between processors to see which processor is busy

65
Linux Unique Thread Architecture
Other OS’s have a concept of Light Weight Processes (LWP) Linux 2.6 everything is a thread, and they are self
(threads) where the parent process spawns owns, and managed. They keep track of their own state and don’t
manages the threads. rely upon a parent to manage them. Linux CAN
simulate LWP to the outside world and this is what you
Think mamma black bear protecting her cubs will see in the demo/labs.
Think busy bees – once born

they are off on their own
Main Process Parent Code
Thread A Thread B Thread C

66
Linux Process to Thread Relationship
So a Linux process is like a beehive that keeps all the common data/honey. The threads
are like the bees that go off and do their work and bring back data/honey where it is all
shared between them.

• Processor state
• Kernel threads
• File system
• Signals

67
Clone() – Threads and Sharing
This system call is the heart of threads and sharing. Depending on what the programmer
tells the thread to share, the thread could act like a full blown HEAVY process (old days),
or like a little lite worker bee that doesn’t carry any baggage with them (Linux today). Other
Unix’s are starting to implement this in some fashion but Linux was first.
int __clone(int (*fn) (void *arg), void *child_stack, int flags, void *args)

68
Thread Summary
• Threads can have 3 contexts:

• User
• Kernel
• Interrupt (very special type of thread, very limited)
• Threads are the workers and the data is ‘usually’ stored in a shared common
process space
• Threads enter kernel space and take on kernel context: virtual memory space,
and security context.

69
Linux Kernel Basics
• Linux Overview
• Threads

70
Interrupts
The kernel interacts with external devices such as network cards in two ways:
1. Interrupts – Only interrupt the kernel when there is something to do
can be more efficient
2. Polling – Continuously see if there is any activity on the device. Very
inefficient and slows down the kernel
Modern Linux device drivers use a combination of both (New API - NAPI). They wait for an interrupt to
occur and begin processing. They will then disable the interrupt and go into a
polling mode until there is nothing more to do (no more data). They will then
re-enable the interrupt to be alerted for more activity.
Interrupt Handler
Kernel
Poll
Hey WAKE UP

71
Interrupt Priority
Interrupts have a priority system. Interrupts such as the system clock
have to be processed right away and are a higher priority. They will stop
all processor activity (even lower level interrupt handlers) to be handled.
Lower priority interrupts will disable its own interrupt line (eg. Network card)
so that it doesn’t get interrupted from the same source and run uninterrupted
(unless a higher priority interrupt comes along).
Kernel
LOW PRIORITY
HIGH PRIORITY
System Clock
http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 493 NAPI

72
Advanced Programmable Interrupt Controller (APIC)
APICs are the hardware that manage interrupts. A motherboard had one I/O APIC that
interfaces with the hardware and talks to LOCAL APIC controllers embedded within the
CPU.
In SMP environments where IRQs can be handled by multiple CPUs APICs can be
dynamically programmed by the kernel to direct IRQs to a specific CPU for balancing out
the handling of IRQs from external devices.

73
Top/Bottom Half Interrupts
Interrupt processing is split into 2 parts:

1. Hardware Interrupt (Top): Device specific, stops processor, has to be quick
2. Software Interrupt (Bottom): Thread-like, heavy lifting when CPU has time
Software Iinterrupt (Bottom ½)

Heavy lifting
Stop processor (Top ½)

Quick and get out
Schedule SW Interrupt: When processor has time

do generic processing of packets.
HW Interrupt: Run device specific handler

74
‘top’ view of interrupts
Hardware interrupt handles Software interrupt handles

device drivers
‘top’ – ‘1’ for CPUs
http://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468/ref=sr_sp-atf_title_1_1?s=books&ie=UTF8&qid=1389542475&sr=1-
1&keywords=linux+kernel+development Page: 3322
75
Interrupts are not Process/Threads
This is why you can’t see interrupts in a ‘ps’ or a ‘top’, different data structure than
processes/threads. Very much like them but CANNOT SLEEP/BLOCK!
Soft Interrupt Table array

Softirq[0] Softirq[1] ……… Softirq[31] Process/Thread Table
(used for ‘ps’ and ‘top’)
Hard Interrupt Table linked list

76
Device Drivers
A network device driver runs within the kernel and has two primary functions:
User Process
Kernel
Driver Transmit Functions Interrupt Handler
Device Driver
1) Organize data so 2) DMA data to kernel

NIC can grab it memory and send
interrupt when ready
77
DMA vs CPU copy
Note that a Direct Memory Access (DMA) processor is responsible for transferring
data between the NIC and the host. This allows the CPU to parallel process other
activities while the transfer occurs. There my be some bus contention between
the CPU and DMA, but not as bad as if the CPU had to perform the transfer.
User Process
Kernel
DMA Interface
Host
Initiate Data DMA chip
Transfer!!!

78
Packet Journey – User to Device
Linux Kernel Net I/O Kernel Thread Linux

On Behalf Of User
Call to device driver to put Copies into DMA space and
User
Send device interrupt
data into the right massages data through Space
structures for the NIC and TCP/IP stack
when data is ready to be tell the NIC it can copy.
copied by DMA transfer
send(socket, msg, strlen(msg), 0)

hard_start_xmit()
Application
eth0
TCP/IP Stack
DMA
Space
http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview
http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505
http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow

79
Packet Journey – Device To User
Linux Kernel Linux User Space
Software Iinterrupt (Bottom ½) User process
Hardware User thread
Massage packet through waits for data on
Interrupt (Top ½) Inside kernel
TCP/IP Processing socket
(Stops processor) context
int recv(int s, void *buf, size_t len, int flags);
Signal to continue Application

DMA
Space User
eth0 Space
TCP/IP Stack
IRQ 177
Soft IRQ copies

data to user
space
Big job! Schedule

Device DMAs software
data to kernel interrupt
DMA memory
http://www.ece.rice.edu/~willmann/teng_nics_overview.html#overview
http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-Venkateswaran/dp/0132396556 Page 505
http://www.linuxfoundation.org/collaborate/workgroups/networking/kernel_flow
80
Monitor # of Interrupts per Device
/proc/interrupts keeps track of # of hw interrupts per interface since boot.
Linux will use eth0 as default for network cards until system gets busy then it tries and
re-balance between CPUs (see eth0)

81
Interrupt Affinity
In Linux, you are able to wire an interrupt to a specific CPU. This is called
Interrupt Affinity. Once again this allows data to be cached locally on a single (set)
CPU for the interrupt handler.
Linux will start with CPU 0 handling all interrupts. (?? Does Linux auto balance??)
Interrupt affinity is used by Check Point CoreXL as we will see in the next section.
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-cpu-irq.html
IRQ 32
CPU 1

82
Linux Network Summary
• Interrupts come in 2 halves

• Top – Hardware device driver – Quick and get out
• Bottom – Heavy lifting of data through kernel
• Interrupts are NOT threads or processes. They have many things
in common but you can’t view them and are internal to the kernel.
• Device drivers come in 2 halves
• Transmit – send data to device
• Receive – Usually interrupt driven to pick up packets

83
Linux Evolution wrt
SPLAT/GAIA
Date Linux Release CP
Release
2000 Linux 2.2 R 4.0
Nov 2001 Linux 2.4 NG FP1
Jan 2008 Linux 2.6 R65 HF02
Today Linux 2.6-18cp R77.10
http://en.wikipedia.org/wiki/Check_Point_VPN-1
Linux Tidbits:
GPL (GNU Public License) –
• No custom mods to Linux, they have to be shared with Linux
community as source code. There is a version 2.6.18cp (not sure status)
• Can compile code with GNU compiler and keep source
private. CP uses internal kernel modules to do this.

84
• Overview
• Linux Review
• SecureXL
• CoreXL

85
Why SecureXL??
The classic problem has been at noon when everyone having lunch at their
desk, everyone starts browsing the Internet and thus slowing down business traffic.
In addition as more applications become web-oriented HTTP traffic is
dominating the network.
SecureXL was primarily created to address the web-browsing-at-noon-problem.
Symptom Verify
Highly utilized system ‘top’, ps –o psr,command
High network traffic on 2 interfaces /proc/interfaces
High HTTP traffic SmartLog
Ifconfig is dropping packets, retransmits Ifconfig, netstat -s

86
HTTP 1.0
HTTP 1.0 uses multiple concurrent requests to retrieve a multi-part web

page for the user.
Text
Picture
Text

87
HTTP 1.1
HTTP 1.1 uses a single request to retrieve a multi-part web

page for the user.
Text
Picture
Text

88
HTTP processing
SecureXL is smart enough to know that all packets after packet #1

are all going to the same site and will hit the same rule.
So SecureXL speeds up the processing of these packets by bypassing
the full rule processing and just relying on state tables built by packet #1
to send the packets through the firewall.
Text
Picture
Text

89
What does SecureXL accelerate?
Secure XL can accelerate (bypass full rule processing) in two

instances:
1) Subsequent connections to the same dest and port
2) On a single connection, packets 2 thru N.
Text
Picture
Text

90
NGF Process Mapping
SND does the acceleration and packet distribution. NGF has 1 SecureXL module to
accelerate network packets. Firewall dispatcher sends packets to the right firewall
instance (there can be many..next section)
https://downloads.Check Point.com/fileserver/SOURCE/direct/ID/7513/FILE/CoreXL_Advanced_Configuration_Guide.pdf
User mode
fwk
Kernel mode
Firewall Dispatcher(fwkdrvr)
SND Performance Pack Packet Handler
(SecureXL acceleration)

91
VSX Process Mapping
In VSX, SND is still one module but SND is aware of individual VSs. SecureXL can
be turned off/on per VS but SND shares info about all VSs when making
acceleration decisions for the whole chassis.
When Check Point moved the fw kernel from the Linux kernel to User Mode, they
left only a little bit of code to work with the firewall dispatcher in place. Other than
that it was a clean compile of the User mode kernel…This was not a massive rewrite
VS0 VS1 VS2
User mode
Kernel mode
Firewall Dispatcher (fwkdrvr)
SND
VS0 SecureXL VS1 SecureXL VS2 SecureXL

92
NGF Firewall Kernel Modules
Here you can see the kernel modules for NGF vs VSX. Basically the same firewall module
is used for both. But…probably…VSX only uses the bottom half processing because the
firewall itself is in user space.

93
SecureXL Acceleration
There use to be a hardware acceleration device by Nokia, now its in software. CP
recently put it back in hardware
SecureXL acceleration
1) Subsequent packets from a
single connection
2) Subsequent packets from
the same source IP, same dest
IP and same dest port
(multiple HTTP requests to
same dest)
94
Packet Journey Thru SecureXL
Linux Kernel Linux User Space
Concurrent processing
of SI’s is possible unlike
Device driver hogs HW interrupts. Can be
Hardware processor. Can’t be interrupted.
Core 3
FW
Interrupt interrupted. Just Instance 3
(stops whole processor transfer data.
single thread)
SND
Device FW FW
Interface
Driver Instance 1 Instance 2
eth0
eth0 Software Core 0 Signal Core 1 Core 2

Core 0
Interrupt ‘Continue’
IRQ 177
Eth0 needs
service
Can’t accelerate
Big job! Schedule this, send on to a
software specific FW SND picks instance
interrupt Instance 1 to process packet
Standard Linux Processing Core XL

95
Firewall Chain – SecureXL Modules
Once packets make it through SecureXL, these are the modules in the
firewall chain that build the SecureXL connection tables and sync with
the SecureXL module itself.

96
SecureXL and IPS/AV/Bot Integration
When IPS/AV/Bot is enabled, not all traffic can be accelerated because it has to be inspected
by the IPS engine.
PSL is the Packet Streaming Library for re-assembly

of IP packets so the IPS/AV/Bot can look for signatures.
SecureXL can forward packets directly to the PSL

and bypass firewall processing. This is called
‘Medium Path’ because it bypasses rule checking on
the 2nd+ packets
Fwaccel stats – SecureXL statistics for Medium Path

97
VSX Detailed Internals
Firewall in user
Packet stays in pool VS2 mode inspects and
Should I creates pass/fail
Accelerate? VS1
Packet Data
HW Interrupt Pool VS0
Construct The Packet
Packet Meta-Data Pointer to packet
SND Queue is what moves
fwk
Outbound to another VS
Fwkdrv in Fwkdrv out

Write pass/drop to msg q
Accelerate
F2F F2F
Outbound to another VS
Dispatches to right IP Stack

VS
Inbound
Message Queue
SND Fwdrv deq
Outbound Legend
Implements action
from fwk to Kernel Code
Device Driver pass/drop
User Space Code
Shared Memory
98
SecureXL Demo - VSX
• Pass 2 gig file thru a firewall

• R75.40VS VSX
• Traffic going through fw-vsx1 a virtual firewall
• All run inside Virtual Box
• No other traffic

99
VSX – SecureXL On
SecureXL runs as a software interrupt within the Linux kernel. You can
see the %si get higher when it is busy.

100
VSX – SecureXL off
When SecureXL is OFF, you can see the fwk1_dev thread handle the
work.

101
AutoBalance Interfaces
Note how the interrupt handling got shifted from CP0 to CP1. More on this later

102
SecureXL Demo - NGF
• Pass 2 gig file thru a firewall

• R75.40VS NGF Standalone gateway
• All run inside Virtual Box
• No other traffic

103
SecureXL OFF – FW busy
Worker threads are idle, SND and FW sharing the work
Busy kernel threads

104
SecureXL OFF – FW sees packets
Firewall sees all packets in fw monitor

105
SecureXL ON– FW idle
Worker threads are idle, SND is doing all the work
Work done in
Work done in SND
SND

106
SecureXL ON– FW no packets
The first packet may/not be seen if its in the state table already. But no others

107
4 types of acceleration
1. Standard state table We talk about these

2. Accept connection templates (e.g. like HTTP)
3. NAT templates – Perform NAT in SecureXL and not in the firewall kernel (sk71200)
[Expert@HostName]# echo 'cphwd_nat_templates_support=1' >> $FWDIR/boot/modules/fwkern.conf

[Expert@HostName]# echo 'cphwd_nat_templates_enabled=1' >> $FWDIR/boot/modules/fwkern.conf
[Expert@HostName]# cat $FWDIR/boot/modules/fwkern.conf
[Expert@HostName]# reboot
4. Drop templates (sk66402)(R76- HF/R76+Standard) – Drop packets in SecureXL, negation of

rules.
5. Other drops (have to test) SAM, DDOS, SmartEvent/IPS triggered.

CONFLICT: sk33781, sk66402, sk98348 all conflict with one another on this topic. Have to test.
Sk98348 says drop templates are the negation of the security policy and drops are accelerated if it does not
match a rule.

108
Standard State Table
Standard state table tuples will be accelerated. Here you
can see a state table entry. Subsequent packets will get
accelerated.
fwaccel conns

109
Accept Connection Temples
Accept connection templates that will be accelerated. Here you can see
where subsequent connections from the same source IP will be accelerated.
We created multiple SSH
Sessions through the firewall same
Client->Server Our demo had 1
accept template

110
fwaccel stat
This command provides the status of SecureXL
State table
Accept
connections
DOS
drops
NAT in SecureXL

fwaccel stats
This commands provides you statistics on which packets are accelerated

and which are forwarded To Firewall(F2F) for rule processing.
Total acceleration
“C” Current Counts
# from accept
templates
NAT performed by
SecureXL
PXL: PSL + SecureXL
IPS packets
Connections sent to
firewall, NOT
XL/SLOW PATH

112
Debug Tips
• If you have any random issues, immediately turn off SecureXL to determine
if there is a difference
• Using ‘top’ to monitor performance, turn SecureXL on/off and see what %SI is
doing
• Might have to distribute SecureXL across multiple cores if %SI is busy and doesn’t
autobalance. See next section.
• Monitor stats to make sure both state table and connection templates are being used
• Move HTTP 1.0 type protocols to the top of the rulebase so they get hit
• Avoid protocols that disable connection template acceleration (more on this at end)

113
SecureXL Licensing
• ADN – Advanced Data Networking & Clustering (formerly ACCL)

• SecureXL
• Dynamic Routing
• CoreXL????????
• ClusterXL
• QoS,load balancing, ISP redundancy
• Looks like CPSB-ADNC
• Platform
• Appliances – All inclusive
• Open Platform
• Ala Carte List - $1500

SecureXL Summary
• Definitely required for high usage gateways

• Easy to administrate (on/off)
• Understand the difference between state table and connection templates
• All takes place in the %SI under ‘top’

• Overview
• Linux Review
• SecureXL
• CoreXL

116
Why CoreXL
CoreXL allows you to utilize multiple processors on a single chassis for
concurrent processing of firewall requests in order to expand capacity
and reduce latency on your existing platform.
Cheaper to expand capacity on a single bigger chassis than to cluster
multiple smaller chassis (ClusterXL)
• ClusterXL: More expensive to purchase
and maintain
• Cost efficient
• Easier to manage
• Easier to debug

117
CoreXL Manages 4 Affinity Types
3) CP Process
VS1 VS2
2) Fw kernel 4) Linux
Affinity Instance
Process
Affinity
FWD syslogd
logging
1) Interface SND
Secure
Affinity
Network
Distributor

118
CoreXL
• Interface Affinity
• Instance Affinity
• Process Affinity
• Linux Process Affinity

119
The LIE
• Interface affinity is grey zone. Could be included in both SecureXL and CoreXL
• Interface affinity can be used with SecureXL license and no CORExl license
• Interface affinity can also be used without CoreXL or SecureXL license,
it is a Linux function
• Here - Interface affinity is grouped with CoreXL for completeness and
topic flow

120
CoreXL Interface Affinity
Interface ‘Affinity’ is the mapping of processors to interfaces to handle network packets.
Default is ALL, which in reality is Core 0.
SND is responsible for managing interfaces assigned to that core. If there are multiple
CPUs handling different interfaces, then each CPU has a different SND.
1) Interface Secure Network Distributor

Affinity SND
• Process network traffic
• Accelerate
• Distribute to firewall
instances

121
MultiQueue (sk80940)
MQ enables assigning multiple interrupts per interface. Certain interface cards have
multiple TxRx queues per interface. Src/Dst flows are tied to a queue. Then queues
are assigned IRQs and tied to specific processors. This technique optimizes CPU
cache utilization.
https://greenhost.nl/2013/04/10/multi-
queue-network-interfaces-with-smp-on-linux/
Standard on R76 and R77

Previous 71.50
IRQ 2 IRQ 3
IRQ 1 IRQ 1

122
MultiQ Restrictions
• R77.10
• Only on appliances…needs the right hardware and drivers
• Supports increased throughput, not so much increased number of sessions
• Based on src/dst assigned to a CPU. So a single high throughput src/dst will
only use 1 CPU and not take advantage of multiple CPUs.

123
Display Interface Affinities
In this complex environment, SND can

concurrently run on cores 0-7

124
Monitor Hits on SND
/proc/interrupts keeps track of # of interrupts per interface since boot. We can use it to monitor
invokations of SND
In this simple environment SND has ability to concurrently run on any core (that isn’t running
a fw instance), but by Linux default it chooses to run on CPU 0. (Probably not good because
all interfaces and processes will use CPU 0).
So if you have interfaces that are dropping packets, you might want to check this if CPU 0
is busy. POINT: Even though the configuration seems balanced, you need to verify!
SND on core 0 is doing ALL the work

for all interfaces
SND has never autobalanced,

all on Core 0
126
IRQ to CPU mapping
Linux maps IRQs to CPUs in the

proc/irq directory in the smp_affinity
file
These are used to program the APIC

(interrupt controller chip). By default
a IRQ can run on any processor but
Linux chooses Core 0
Why core 0? Well the kernel doesn’t

want to have the interrupt context
float between CPUs. This way it keeps
interface interrupt data in local cache.
R77.10 changed from ‘all’ to CPU 0.

127
Setting affinity command

128
Setting interface affinity
sim affinity –s sets the interface affinity by changing the values in the smp_affinity
file. This in turns programs the APIC to send interrupts to a different processor.
NOTE: Interface settings will survive reboot (BUT not CoreXL settings (next)).

129
Impact Immediate
No reboot, sim affinity –s takes effect immediately

130
Distribute SND processing
This is how you distribute SND
processing on interfaces that are overloaded and
dropping packets. Give those interfaces/SND VS1
their own processor that does not have other FW
components on it
VS2
SND SND
Busy SND Secure

Network
Busy Distributor

131
SND Chooses non-fwk CPUs
When an interface is set to All, it will attempt to use a CPU that is NOT
being used by a firewall instance. But it will try to use a CPU that is being
used by another interface…in order to keep the the local CPU cache fresh.
So when under low CPU usage, most interfaces will default to ALL (below) and
be autobalanced as CPU and interface activity picks up. The default for ALL
is CPU 0…until CPU activity picks up
R77, interfaces will be set to the default CPU0

132
Interface Auto - Balance
Look what happens when straining a CPU with a 3 gig SCP transfer
between members directly connected.
Every 60 seconds, CoreXL examines the CPUs to see if they are busy.
If they are busy it will rebalance interfaces to non-busy CPUs.
(fwkernel and Linux processes rebalance every 1-2 seconds).
(NOTE: I do NOT know how to set back to autobalance once you hard set the interfaces
except by reboot on NGF, or factor defaults on VSX)
Below the fw kernel saw the CPU going to 80 si% and rebalanced the interfaces
from ‘all’ to ‘eth1:1’, gave eth1 its own CPU
Before
After

133
Busy box?
How to tell if a box is ‘busy’??? Here is an example of an Internet gateway
with 10gig interfaces. This is the /proc/interrupts table. You can see that
only CPU 0 has been used to process network packets.
This tells us that the CPUs have not become busy since reboot and the fw kernel
has not done any rebalancing. If the ifconfig shows packet drops, then you have
different issues than CPU not being able to handle the load.

134
Set Interface Affinity – No Performance
Pack
$FWDIR/conf

135
CoreXL

136
VS1 VS2
2) Firewall
Instances
FWD syslogd
logging
SND
Secure
Network
Distributor

137
HyperThread Review
Enabling Symmetric Multi-Threading (SMT) or Hyper Threading (HT) doubles the number of logical processors.
• Note the difference between HT, Dual Core and Dual Processor. Note where the caches are! Remember this
when you assign interfaces so you keep the cache hot
• Works just like Linux thread processing where a process (web server) has 2 threads (2 clients requesting
pages) and the kernel can preemptively multi-task the two threads so it seems like they are parallel
processing.
• Without HT. Each Linux thread gets a time-slice by the Linux kernel but only 1 thread runs at a time.
• With HT: At the hardware level there is a mini-Linux like kernel that can multiplex/task 2 threads concurrently.
So the two threads could conceptually start and finish within 1 kernel time slice instead of 2 separate time
slices.
• Performance improvement 30%???
Without Hyperthreading, Managed by Kernel
Thread 1 Thread 2 Thread 1 Thread 2
WithHyperthreading, Managed by HW
http://blogs.msdn.com/b/gauravseth/archive/2006/03/20/555519.aspx
TIME

138
HT and Cache Sharing
Remember that the kernel instances share a state tables. So when allocating
instances keep similar data flows on the same cache so that portion of the connection
table is always in cache.

139
R77.10 Uses SMT/HT
Implemented in R77+
Restrictions:
• Only enhances performance of IPS/AB/AV CPU intensive functions and
NOT I/O operations. Too many interrupts may actually slow it down.
• Supported only on R77+ GAIA
• Only on Check Point Appliances
• Has to be enabled in the BIOS
• Does not work with large number of HIDE NAT connections. Each CPU has
pre-allocated # of HIDE NAT slots. If one CPU uses all its HIDE NAT slots
then it can’t handle new HIDE NAT connections.

140
Assigning Instances to CPUs
• NGF Firewall Affinity

• VSX Firewall Affinity

141
NGF Standalone Instance Affinity
In NGF there is only 1 firewall. With CoreXL, the kernel will replicate itself X times,
depending on how many firewall instances you setup. Each instance will parallel process
network traffic with the SAME shared rulebase and state tables.
Each instance has an ‘affinity’ for a specific processor.

142
NGF CoreXL
In a NGF gateway, CoreXL
generates X copies of the kernel
into individual Linux kernel
threads
Affinity

143
NGF Process/Thread Mapping
Before we assign affinity, we should look at what types of processes/threads we
are dealing with. When multiple processes have the same PID, that means they
are threads sharing the VM of the parent (CPD and FWD below).
Here you can see the firewalls are individual KERNEL threads inside the kernel with a parent
of PID 1 – ‘init’. Kernel threads usually come in thread groups of size 1, unlike user space.

144
NGF Process Spawn Tree
Here you can see how the firewall processes were spawned in what order

145
• NGF Firewall Affinity

• VSX Firewall Affinity

147
VSX CoreXL Affinity
Where NGF has 1 firewall on a chassis, VSX has multiple firewalls running on a single physical chassis.
Each firewall is represented by a Virtual System (VS).
VSX VS’s are a different than NGF. Each VS has a 1 firewall instance that is executed by 1
corresponding Linux OS User mode process (not totally true, but not lying and still making the point).
VS VS VS VS

148
R75.40 VSX Process Architecture
VS instances are implemented with Linux threads (user mode but mapped onto
kernel threads so they can be scheduled by the kernel)
fw kernel VS
=
• CPU Instructions instance instance
• Processor state
• Kernel threads
• Security attributes VS
instance
VS
instance

149
VS instances
The terminology gets a little VS

confusing because CP is not
explicit nor consistent in defining
what an instance is for VSX.
A firewall kernel instance which

runs on behalf of a VS, can also fw kernel VS
instance instance
be further subdivided into VS
instances. These VS instances can
VS
be assigned to individual instance
processors.
VS
instance

150
VSX VS0 CoreXL Affinity
Here you can see we are configured 3 VS instances within VS0 (very important!!).
I have 3 VS instances running. So you can see that VSi0-VSi2 are allowed to run on any
of the 1-3 CPUs. CPU 0 is reserved for eth0 traffic (hold on).

151
VS0 Processes and Threads
So VS0 has 4 VS instances (VSi) generated for it that are watched over by
the watcher daemon.
Processes Threads

152
VS0 CoreXL Config
You can only configure CoreXL VS0 from the command line….
Trying to configure more

VS instances for VS1 from
the command line and
look….

153
Change non-VS0 CoreXL
OK. We’ll listen to directions…
1= CoreXL OFF
2+ = CoreXL ON

154
Look Ma Babies!!!
• VS1 now has 2 instances.
• CPD has 1 main and 2 instances
• FWD has 4 daemons (not sure why)

155
Linux view of VSX instances
VSX uses USER threads for all components

156
VSX Process Spawn Tree
Here you can see the parent-child relationship of how firewall instances are spawned.
Note: the process names change, like fw could be fwd – Linux command line issue.
pstree -p

157
• NGF Firewall Affinity Assignment

• VSX Firewall Affinity Assignment
• Monitor Affinity

159
Default Affinity Assignment
The default affinity for firewall instances is to assign CPUs from high to low. There can never be more instances than
CPUs so that mapping is 1:1. The default affinity assignments for interfaces is to CPUs that are NOT running firewall
instances…if possible.
Note that the default assignment is probably adequate for 90% of the cases….unless you have a lot of busy network
interfaces, help processes, Linux processes that interfere with the firewall instances. For example, if eth0 ran 100% of
CPU0, then you might want to move the firewall instance.
From sk98348
# of Cores # of FW # of SNDs
Instances
1 1 0 (Corexl disabled)
2 2 2
4 3 1 Reserved for
SND
6-20 # cores -2 2
21-30 # cores -4 4
160
NGF: Assign Affinity
When you re-assign an instance to a CPU, you are telling the instance to only
use THAT CPU…when the CPU is free. So double-edge sword:
1. GOOD: Guarantee the cache will be always hot on that CPU
2. BAD: What if that CPU is busy with other assigned processes…Has to wait till
end of the other process timeslice to get CPU time. You could up its priority.
So make sure you choose a CPU that is NOT assigned to any other process if
possible.

162
Assign ALL CPUs
You can set affinity to ALL CPUs by assigning all CPUs to the instance.
You can obviously also set the affinity to a subset of CPUs.
Note: that you forfeit hot caching

163
Affinity Set via Linux
Note that instead of fw ctl affinity command, you could just use the
regular Linux affinity command ‘taskset’. Does the same thing.

164
Permanent Affinity
Custom firewall instance affinity are not permanent (but INTERFACE affinity is).
On reboot you have to re-assign affinity. You can make firewall instance affinity
permanent configuring it in $FWDIR/conf/fwaffinity.conf.
NOTE: auto vs. all

165
Stickiness
State table tuples are ‘sticky’ to a single core. Once you start a network
conversation through a specific core, the associated tuple will always use
the same CPU to process that network conversation.
So a backup process between backup server A and client B for example
could monopolize a single CPU.
<1.1.1.1, 1111, 2.2.2.2, 2222, TCP>
2.2.2.2 1.1.1.1
CORE 0

166


167
VSX Default Affinity
The default config for VSX is firewall instances are assigned to all but CPU 0
(reserved for interfaces). Default varies depending on components, but it won’t be
“ALL”.

168
VS Affinity Command Hierarchy
In VSX there is a hierarchy how VS affinity commands work. You can apply the instance
command to the ‘V’ entire VS, ‘P’, only the firewall processes or ‘I’ a specific VS instance.
The ‘I’ then inherits from the ‘P’ which inherits from the ‘VS’.
‘VS’ for Virtual System ‘P’ for Firewall Processes

‘I’ for instance
cpd
fwk
VS
fwd instance
vpnd
VS0
169
Impact of Setting Affinity
The SRC column shows at what level the affinity command for the process
was issued. ‘V’ means the command was issued to ALL the components
of a VS. ‘P’ means only to the firewall instances. ‘I’ means a single fw instance.

170
VSX set affinity command
Here is the command for setting affinity for the 3 levels of processes.
Note that if you do not set a affinity for a level, the level will inherit
the affinity from the previous level.

171
VS affinity config files
After you set affinity you can see the impact in these configuration files. Because
of these files, VSX affinity is permanent…unlike NGF!!!
Here are the affinity configurations that the VSs use to set their affinities. As
you set affinities at the different levels, these files will begin to appear.
This is how a VS instance knows what affinity to use. If there is no config file
at the I instance level, then it goes up to the P Process level config file, etc.

172
Set VS affinity
Here we set the VS affinity. Note that it sets the affinity for ALL processes in VS0.
Firewalls, Firewall Helpers, Linux processes.
Note how the SRC column is “V” for the VS affinity configuration file

173
Set P Process Affinity
This will set the P Process affinity for JUST the fwk processes in a VS.
NOTE: the ‘P’ means to set affinity to what is found in the ‘P’ config file

174
Set P Process Affinity
This will set the P Process affinity is for JUST the fwk processes and VS instances.
NOTE: -fwkall will set the fwk VS instances for ALL the VS’s.

175
Set ‘I’ VS Affinity
Here you can see we set he ‘I” VS Instance affinity
The “I” in the SRC column means the affinity config comes from the ‘I’
config file

176
Now let’s set everything to ALL
As with NGF, if you set affinity with all CPUs, it will be set to ‘ALL’. Note it only
sets the affinity for VS0, and not the other VSs

Proprietary and Confidential: No part of this document may be reproduced without permission from 177
VSX: CoreXL set per VS
CoreXL can be enabled/disabled per VS instance, just like SecureXL
VS0 use cpconfig, VS1+ use SmartDashboard to set CoreXL to 1

178
Reset to Defaults
REBOOT!!!!!

179
Changing Cluster CoreXL config
At boot - Cluster membership is tested to ensure same # of CPUs in CoreXL.
When changing CoreXL cpconfig CPU configuration in a cluster:
1. Start on standby member B, bring it down

2. Change number of processors - cpconfig
3. Reboot
4. Member B Comes up in the READY state
5. Fail over Active member A to the Ready member B (Stateless)
6. Modify formerly Active member A
7. Reboot member A
SETTING AFFINITY (not CoreXL Firewall Instance count) must be done manually in both members
and does not impact cluster status
Member A - Active Member B - Ready
https://sc1.Check Point.com/documents/R76/CP_R76_ClusterXL_AdminGuide/7298.htm
180
Cluster XL Hints
1. Keep SecureXL and CoreXL configuration the same on both members

2. On upgrades, config will carry over but not necessarily on from scratch
migrations. ( e.g. process assignment)

181
Summary: NGF to VSX CoreXL
NGF can delegate its firewall processing VSX has each VS represented by a Linux process with 1 or
into multiple fw instances. Each fw instance more internal VS instances mapped to Linux threads.
is represented by a Linux kernel thread The whole Linux process OR the VS instance (thread) can be
and can be assigned a core affinity. mapped to a processor
Linux
Process Linux Firewall Instance
/Firewall Process
Instance /Firewall
Instance
Firewall Instance Firewall Instance
Linux
Process
/Firewall Linux
Instance Process
/Firewall
Instance



183
‘top’ Tricks for CoreXL
F SORT
j Processor
u Number of page faults
f Add
columns
j Processor
u Number of page faults
1 Show all processors
i,z Just show running processes
c Command vs process name

184
Important TOP Stats with ‘f’
command

185
Unix ‘top’
Show all processors and see what process is running on what processor

186
Monitor Running Processes
With ‘f’ ->‘i’ and ‘j’ and ‘u’ you can monitor running processes that might be
faulting or jumping processors

187
CoreXL ‘ps’ arguments
-e All processes
-L All threads
-o Specific columns (below)
pid Pid of process
psr Processor id
cmd,comm,com Command/Process names in long-short formats. with

mand,args (args) arguments
lwp Thread ID
majflt Page faults

188
Unix ‘ps’ to monitor processor
‘watch –d –n1’ will execute a command every 1 second and highlight changes
‘ps’ can be used to monitor processor usage by a specific process

189
‘watch’ Interrupt table

191
CoreXL

192
3) CP Process
VS1 VS2 4) Linux
Affinity Process
FWD syslogd
logging
SND
Secure
Network
Distributor

193
List firewall helper daemons
Firewall helper daemons (cpd, fwd, vpnd) are running in the VS0-VSX VS’s.
Linux processes are running under the control of VS0.
Many times you will see they are able to run on ALL processors
Linux only processes

194
Set Process Affinity
Let’s say your fwd logging process is monopolizing the system with heavy
logging. You an assign a whole processor to it to handle the load.
NOTE: /opt/CPsuite-R75.40VS/fw1/conf/vsaffinity_exception.conf are a list of LINUX processes that are not impacted by
the affinity command. You have to edit this list to modify LINUX process affinity

196
Process to CPU
Busy processes such as VPND or FWD (logging) can be dedicated
to a single CPU because they might dominate a core or swap cores
frequently thus losing their cache freshness.
Note in VSX it is specific to your VS context
FWD
Logging/HA

197
CoreXL Licensing
• Appliances include all the cores

• Open Servers
• You can select how many cores to buy
• CPSG-PCC
• CC = # of cores
• CPSG-PCCBB
• BB = # of blades
• CPSG-P1207
• Security Gateway
• 12 core
• 7 blades
• NOTE: You only have to license the cores that FWK run on. You can
buy a 2 core license and have a 12 core gateway. ..10 of the cores
will run Linux, SND, CP helper programs. 2 cores run FWK
• Example: Ala carte 2 core to 4 core upgrade $7500 list

• Overview
• Linux Review
• SecureXL
• CoreXL

199
Tuning Tips
• Traffic to Rules to SecureXL to CoreXL tuning

• VSX: Keep traffic flows on same system
• VLAN resources
• CPU or Memory
• IPS Integration???
• Suggested CoreXL Configurations
• Command summary

200
Classic Target
2 interface Internet firewall, busy at noon with HTTP traffic interrupting

business traffic
Symptom Verify

201
Determine Your Traffic Patterns
eth0 to eth3
Use SmartLog to determine your major traffic flows

between interfaces.
Also look for busy tuples, remember they are sticky to
processor!!

202
SecureXL Restrictions
The following traffic is not throughput (state tables), nor connection-rate (accept
templates)accelerated by SecureXL:
Traffic types other than TCP, UDP, PIM, GRE, ESP
First packets of any new TCP session, unless a "template" exists
First packet in a UDP session
Traffic matching certain Firewall rules:
rules with a service that uses a resource
rules for dropping or rejecting traffic
rules where the source or destination is the gateway itself
rules with a Security Server
rules with user authentication
rules with session authentication
The following traffic is not connection-rate (accept templates) accelerated by SecureXL and will stop
building templates in the rulebase if they are found:
Non-TCP/UDP connections such as PIM, GRE, ESP ---- ICMP
Protocols that are not connection intensive such as SMTP, FTP, RPC, NFS, NNTP, NTP
Complex connections such as IPSec VPN, FTP, H.323, etc.
Traffic in environment using NAT (for security, NAT addresses can change and can be shared)
203
Check SecureXL and Rulebase
ICMP prevents SecureXL Accept Connection templates from accelerating HTTP 1.0/type
connections. Need to move to end of rulebase.

204
HTTPS and CoreXL tuning
Use /proc/interrupts and ifconfig to see if any of your interfaces were
struggling to keep up. If the interfaces are constantly rebalancing or
dropping packets, then you might have a problem.
All interfaces
on eth0!

205
Processor Utilization
Look to see if CPU 0 is busy.
Busy FWD
Busy core 0

206
Balance CoreXL
So investigation shows Core 0 is busy. This means you should dedicate a ‘free’
processor to eth0 because eth0 is doing a lot of work with regular processes AND
interface handling
Give eth0 and eth3

their own core
Then give the offending process its own processor.

207
Drop Rule
If SmartTracker or SmartEvent shows a Denial of Service or heavy traffic from a
malicious source, consider using drop templates or Suspicous Activity Monitor
instead of the rulebase to drop
the traffic. The drop templates are enforced by SecureXL and done before you
have to go through the rulebase.
https://supportcenter.Check Point.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk67861

208
Tuning Tips

• VLAN resources
• CPU or Memory?
• Command summary

209
VSX Archtecture
With VSX which is the most efficient way to route

packets between two networks and why?
Router
Router
VSX VSX VSX VSX

Chassis Chassis VRouter Chassis Chassis
VRouter VRouter
11 1 1

VSX Architecture Design
Balance VSLS VSs based on traffic flow w/ SecureXL caching

If you have lots of traffic between two VSs, then keep them on the same
gateway because SecureXL /CoreXL will cache states and accelerate locally
VSX VSX
Traffic states cached Chassis Chassis
locally and SecureXL will
not send it through user
mode kernel.
CoreXL will cache states in
processor CPU dedicated
to those interfaces
SND SND

Tuning Tips

• VLAN resources
• CPU or Memory?
• Command summary

212
Bonded(VLAN) Interfaces
When you have a bonded VLAN that trunks multiple networks over a single
logical interface, (assuming it will be heavily utilized) considering assigning
a dedicated CPU to each interface to handle the load and keep the caches
‘fresh’.
VLAN
VLAN
VLAN
VLAN
Core 0 Core 1
Tuning Tips

• VLAN resources
• CPU or Memory
• Command summary

214
CPU or Memory
How do you know if you need more CPU or more memory or

both?
? ?

215
‘top’ is WRONG
Linux 2.6.18 ‘top’ is just WRONG. It does not compute SWAP space correctly
because it does not take into account (e.g.) that the processes is sharing 2GB
of space from a shared library along with 10 other processes. So each of the 10
processes thinks it alone is using the 2GB of shared space.
Below you can see that nautilus VIRT is 433MB, and SWAP is 427Mb, but so are
most the other processes. It just doesn’t add up to the ‘used’ SWAP.
So its hard to tell what processes are using what memory and what is using SWAP space.

216
#1 Mo Memory
Because you can’t trust ‘top’ too much, a better measure is to look at the major
page faults for running processes. If they are faulting a lot, you have a problemo.
Using ‘top’ watch the running processes by typing ‘i’. This will only list the running
processes. Then use ‘f’ and ‘u’ for listing page faults.
Below is a real live MLM that is faulting heavily on one of the logging daemons. You can
have a fast CPU and add more CPUs, but fwd will spend most of its time swapping pages.

217
#2 More CPU
If you have a lot of active processes that are NOT faulting, then you have 3
options:
1. More CPUs
2. Faster CPUs
3. Set affinity on the most busiest processes so they are using their hot caches
and not spending time flushing caches
4. Set the affinity of the busiest processes to -15 so that they get a bigger chunk
of the timeslice

218
#3 Memory Leak
1. If ‘Swap used” is > 0 and growing, you may have a memory leak.
I
2. If “VIRT” is growing over time on a process, you may have a memory leak.

219
Tuning Tips

• VLAN resources
• CPU or Memory
• Command summary

220
Classic Target - NGF
2 interface Internet firewall, busy at noon with HTTP traffic interrupting
business trafficAssumes that SecureXL is working on ALL rules and SND is doing
most the work and not the FWK. NO IPS/AV/ThreatPrevention,etc.
Goal is to allocate evenly. Obviously this is only a guess, but you’d have
to evaluate your system with the following commands to have more accurate
measurement
Symptom Verify

221
Balancing Goals
This is the take-away slide, the value behind the whole course.
Allocate CPUs in the following priority order:
1. Share cache for common data

2. Allocate CPUs to busy Internal interfaces
3. Allocate CPUs to slower less busy External interfaces
4. Allocate CPUs to FW instances
5. Remaining threads are usually idle so distribute evenly
and let kernel find idle processor

222
Reminder: Share Cache!!

223
Suggested CoreXL Distributions-NGF
Type CPU/Item 1 2 4 8 (2 core) 12 (2 core)
FW cpd 0 All All 3 3

Processes
fwd 0 All All 3 4
Linux Linux All All All All All

processes
Interfaces External 0 0 0 0 (cache 0) 0 (cache 0)

(SND) Interface
Internal 0 1 1 1 (cache 0) 1 (cache 0)

Interface
Sync 0 All All 2 2
Fw kernel fw_0 0 0 2 4(cache 1) 6(cache 1)

instances
fw_1 1 3 5(cache 1) 7(cache 1)
fw_2 6(cache 1) 8(cache 1)
fw_3 7(cache 1) 9(cache 1)
fw_4 10(cache 1)
fw_5 11 (cache 1)

224
Classic Target - VSX
VSX with multiple business silos to Internet. Assumes that SecureXL is working
on ALL rules and SND is doing most the work and not the FWK. NO IPS/AV,etc
NOTE: fw0 is not handling any traffic aside from sync. Only mgt traffic
Goal is to allocate evenly. Obviously this is only a guess, but you’d have
to evaluate your system with the following commands to have more accurate
measurement
Symptom Verify
Highly utilized system ‘top’, ps –o psr,command,

vsx resctrl, vsx memstat

225
Suggested CoreXL Distributions-VSX
Type CPU/Item 1 2 4 8 (dual core) 12(dual
core)
Interfaces Ext VS0 0 All 0 4 6
(SND)
Ext VS1 0 0 1 1(cache0) 0 (cache0)
Int VS0 0 All 0 4 6
Int VS1 0 0 1 0(cache0) 3 (cache0)
Sync 0 All 0 4 7
Fw kernel fw_0 0 All 0 4 8

instances
fw_1 (VS1) 0 All 1 5 9
fw_2 (VS2) 0 All 2 6 10
fw_3 (VS3) 0 All 3 7 11

226
Suggested CoreXL Distributions-VSX
Type CPU/Item 1 2 4 8 (dual core) 12(dual

core)
cpd(VS1) All All All but 0-3 6,7,8
fwd(VS0) All All All but 0-3 6,7,8
Linux Linux All All All but 0-3 All

processes

227
Tuning Tips

• VLAN resources
• Command summary

228
SecureXL Cheat Sheet
sim affinity –l List interface affinities

NGF: fw ctl affinity –l –v -a List all affinities including
interfaces
VSX: fw ctl affinity –l -x List all affinities including
interfaces
sim affinity -s Set interface affinities
fwaccel stat Review SecureXL status
fwaccel stats Review SecureXL stats
fwaccel conns Review SecureXL state table
fwaccel templates Review SecureXL accept
connection templates

229
CoreXL Commands
List interface affinities

NGF: fw ctl affinity –l –v –a List all affinities including
fw ctl multik stat interfaces
VSX: fw ctl affinity –l –x List all affinities including
fw ctl affinity –l -x –flags tkn interfaces.
fw ctl multik stat -flags tkn lists internal threads
cat /proc/interrupts List IRQ table and interrupts
handled by CPU
watch –d –n1 “ps –ef | fgrep Watch how a process changes CPU
syslogd”
top -------- F->j
fw ctl affinity –s –d –fwkall 3 Set all process affinities to ALL
(auto)

230
Thanks!!!!

231

ProxySG SWG Initial Configuration Guide

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

ProxySG SWG Initial Configuration Guide

Загружено:

Авторское право:

Доступные форматы

Balancing CoreXL and SecureXL

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

SND • Processing incoming traffic from the network interfaces

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

• More expensive to purchase

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

NGF (Next Generation Firewall) is a

FWK VS0 VS1 VS2

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

• Secure Platform (SPLAT) is the commands and kernel modules added to

GAIA command shell – Self

SPLAT Linux command set

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Describes many topics in these articles. After this

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Allocate CPUs in the following priority order:

1. Share cache for common data

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

/proc file system

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

CPD Push Policy FWM

Smart Center MGT Station

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

cpd cpd cpd

fwk fwk fwk

fwd fwd fwd

vpnd vpnd vpnd

User mode VS0 VS1 VS2

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

User mode VS0

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Eth0:2: 10.2.0.101/24 Eth0:1 : 172.17.0.111/24

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

• Some tried to simplify the

• Original kernels were monolithic:

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

1. Scheduler – Improved fair scheduling with 100’s of processors

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Most kernel work done

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Program becomes a Process

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Saved Frame Pointer

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

If the programmer forgets to free the memory, then you have

Sometimes heap space becomes so fragmented with mixed

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Saved Frame Pointer

Program Counter New PC

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

NOTE: Most programs don’t use all 4 gig at

© 2013 Midpoint Technology, Inc. 952-837-6206 – sales@midpointtech.com

Here in my ‘memgrow’ program I allocated 1GB of HEAP memory, but didn’t