Expedition - Security Policy Optimization

via Machine Learning

Professional Services - Security Management Framework
PAN-OS 8.1
Contact Information

Corporate Headquarters:
Palo Alto Networks
3000 Tannery Way
Santa Clara, CA 95054

About This Document

The Operational Enablement documents are designed to enable and inform customers on how to
manage Palo Alto Networks technologies in a consistent and efficient manner. These documents assume
that the reader is already familiar with Palo Alto Networks technology and they are meant to serve as
sections within a runbook on how to manage the platform once deployed.
These documents do not replace other technical documentation published by Palo Alto Networks on their
products and features. For more information about anything referenced in this document, see the
technical documentation found at:
https://www.paloaltonetworks.com/documentation

© 2019 Palo Alto Networks, Inc. Palo Alto Networks is a registered trademark of Palo Alto Networks. A list of our
trademarks can be found at http://www.paloaltonetworks.com/company/trademarks.html. All other marks mentioned
herein may be trademarks of their respective companies.

Expedition Security Policy Optimization 2

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Table of Contents
Contact Information .................................................................................................................................... 2

About This Document ................................................................................................................................. 2

Overview ...................................................................................................................................................... 4

Security Rulebase Creation From Logs Operational Guide ................................................................... 4

Rulebase Creation from Logs Procedure ................................................................................................. 4

Expedition Security Policy Optimization 3

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Overview
A firewall security rulebase is a critical component of the overall security architecture. An effective security
rulebase should fulfill two requirements:
1. Support an organization’s business requirements by allowing authorized access to requested
resources, whenever needed
2. Ensure that only authorized entities are allowed to access critical resources, based on the least
privilege principle.
In order to implement a security rulebase, firewall administrators need the system’s owners to provide a
communication matrix describing the communication flows that are required in order to support their
business requirements. However, more often than not, the communication matrix is either not available,
or is loosely defined, which results in the implementation of a permissive security policy.
This document describes a methodology that can be used in order to tighten a loosely defined security
rulebase by leveraging the Machine Learning (ML) feature introduced with Expedition.

Security Rulebase Creation From Logs Operational Guide

Improves Security X Improves Manageability X
Improves Performance X Improves Availability
Customized Benefit/Gain:

Suggested RACI Role Security Activity Frequency (i.e., Monthly

(generic suggestion Administrator/Engineer daily, weekly, monthly,
which will vary by etc.)
organization)
Required Skills: VMWare; Expedition; PAN-OS

Rulebase Creation from Logs Procedure

Step 1 – Increase Expedition disk space. As logs need to be imported into Expedition instance, there’s
typically a need for increased disk space.
• From the VMware Hypervisor (ESXi, Fusion or Workstation), add a new disk to the Expedition
VM. The VM might need to be shut down in order to complete this step.
• After booting up the Expedition VM, connect to it using SSH and run the following commands to
add the new disk to the volume group and increase the available disk space.
###check the name of the Volume Group. It should display “Expedition-vg”
admin@expedition$sudo vgdisplay
admin@expedition$sudo pvcreate /dev/sdb
admin@expedition$sudo vgextend Expedition-vg /dev/sdb
### request extension of the volume by using 100% of available space
admin@expedition$sudo lvextend -l +100%FREE /dev/Expedition-vg/root
### (notice /dev/mapper/Expedition—vg-root) is 18.GB of size
admin@expedition$df -h
admin@expedition$sudo resize2fs /dev/Expedition-vg/root
admin@expedition$df -h
### (notice /dev/mapper/Expedition—vg-root) increased its size

Expedition Security Policy Optimization 4

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Step 2 – Ensure that php is configured to reconnect automatically to the database if connection is lost.
• Connect to the Expedition VM using SSH.
• Execute the following commands to change the value of mysqli.reconnect to On if needed.
expedition@Expedition:/datastore$ sudo sed -i 's/mysqli.reconnect\ =\
Off/mysqli.reconnect\ =\ On/g' /etc/php/7.0/apache2/php.ini
expedition@Expedition:/datastore$ sudo sed -i 's/mysqli.reconnect\ =\
Off/mysqli.reconnect\ =\ On/g' /etc/php/7.0/cli/php.ini

• Execute the following commands to verify that mysqli.reconnect value is set to On and
restart apache if needed.
expedition@Expedition:/datastore$ grep mysqli.reconnect /etc/php/7.0/apache2/php.ini
mysqli.reconnect = On
expedition@Expedition:/datastore$ grep mysqli.reconnect /etc/php/7.0/cli/php.ini
mysqli.reconnect = On
expedition@Expedition:/datastore$ sudo service apache2 restart

Step 3 – Comment out the bind-address command in /etc/mysql/my.cnf

expedition@Expedition:/datastore$ sudo sed -i 's/^bind-address/#bind-address/g' /etc/mysql/my.cnf
expedition@Expedition:/datastore$ grep bind-address /etc/mysql/my.cnf
#bind-address = 127.0.0.1
#bind-address=0.0.0.0

Step 4 – Import/export the logs from the firewall.

There are different ways to import logs from the firewall:
• Scheduled logs export from firewall. With this method all the logs from the firewalls are
exported on a daily basis using SCP or FTP. You can’t control which specific logs to export (i.e.
all rules logs will be exported), however, you usually only need logs for a specific rules or set of
rules.
• Manual logs export from firewall. Logs can be filtered and exported manually either from the
GUI or from the CLI. You can then filter the logs to export based on the rules that generated them
and the time period. The following command allows you to export logs to Panorama using SCP.
admin@firewall$scp export log traffic query "( rule eq rule1 ) or ( rule eq rule2 ) or
( rule eq rule3 )" start-time equal YYYY/MM/DD@HH:MM:SS end-time equal
YYYY/MM/DD@HH:MM:SS to user@scp_server_ip:/PATH/fw-

• Syslog forwarding from firewall. Firewall can be configured to forward the traffic log to a syslog
server. This method is the one that will be used in our example. The configuration steps for
syslog forwarding for a firewall managed by Panorama are described below.
− From Panorama, under Templates > Device, select the template that is relevant to your
firewall.

− Go to Server Profiles > Syslog and click Add.

− Give a Name to the profile (for example, SYSLOG-EXPEDITION), and click Add under
Server tab.
− Type a Name for the server, enter the Syslog Server IP address.
Expedition Security Policy Optimization 5
via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 − Click OK.

• Go to Device Group > Objects > Log Forwarding. Select the device group that is releveant to
your firewall in the Device Group drop-down box and click Add at the bottom.

Expedition Security Policy Optimization 6

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Enter a Name for the Log Forwarding Profile for example, LOG-FWD-EXPEDITION) and click
Add at the bottom of the dialog box.

• Enter a Name for the Log Forwarding Profile Match List (for example, ALL-TRAFFIC-
EXPEDITION).
• Under Syslog section, click Add, and select the Syslog Server Profile you previously created.
• Click OK.

Expedition Security Policy Optimization 7

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • From Panorama, go to Device Group > Policies > Security > Pre-Rules (or Post-Rules). Make
sure the relevant device group is selected.

• Click on the security rule(s) you need to analyze and go to Actions. Click the Log Forwarding
drop-down box and select the log forwarding profile you created previously.
• Click OK.

• Click Commit and select Commit and Push. Click the Commit And Push button.

Expedition Security Policy Optimization 8

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Step 5 – Configure syslog-ng on Expedition server.
In order to collect the syslog messages the firewall is going to forward to our Expedition instance, we
need to install and configure syslog-ng.
• Log into Expedition instance using SSH, and install syslog-ng using the following command:
admin@expedition:~$ sudo apt-get install syslog-ng

• Create a folder to store the logs received from firewalls and give it the appropriate permissions
using the following commands:
admin@expedition$sudo mkdir -p /PALogs/syslog
admin@expedition$sudo chown www-data /PALogs/syslog
admin@expedition$sudo chmod u=rwx,g=rwx,o=rwx /PALogs/syslog

• Create a syslog-ng configuration file to forward the firewall logs to our newly created folder.
admin@expedition$sudo nano /etc/syslog-ng/conf.d/firewalls.conf

• Copy and paste the following in the configuration file:

##################################################
options {
create_dirs(yes);
owner(www-data);
group(www-data);
perm(0640);
dir_owner(www-data);
dir_group(www-data);
dir_perm(0750);
};

##################################################
source s_udp {
network (
ip-protocol(6)
transport("udp")
port(514)
);
network (
transport("udp")
port(514)
);
};

##################################################
destination d_host-specific {
file("/PALogs/syslog/$HOST-$YEAR-$MONTH-$DAY.csv");
};
log {
source(s_udp);
destination(d_host-specific);
};

Expedition Security Policy Optimization 9

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Save the file by pressing ^O and quit with ^X.
• Restart the syslog-ng process using the following command:
admin@expedition$sudo service syslog-ng restart

Step 6 – Make sure incoming syslog traffic is allowed on the Expedition instance.
• Log into Expedition instance using SSH, and enter the following commands:
admin@expedition$sudo firewall-cmd --permanent --zone=public --add-port 514/udp

Step 7 – Configure Expedition for Machine Learning .

• Log into the Expedition GUI and go to SETTINGS > M. LEARNING.
• In TEMPORARY DATA STRUCTURE FOLDER pane, enter the PATH that will be used by
expedition for the processing of the logs. In our case we will use /datastore.
• Under SERVER INFORMATION, type the IP address of the Expedition instance in the
Expedition ML Address text box.
• Click Save at the bottom right.

• Log into the Expedition instance using SSH to create the datastore folder and assign ownership
and permissions.
admin@expedition$sudo mkdir /datastore

admin@expedition$sudo chown -R www-data /datastore

admin@expedition$sudo chmod +x /datastore

• From the Expedition GUI, go to DEVICES and click the + button on the to right to add the
Panorama device.

Expedition Security Policy Optimization 10

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Type in a Device Name, the Hostname/IP, and Serial #. Select the Model from the drop-down
box and click Save.

• The Panorama device should now appear in the list of IMPORTED DEVICES. Double-click the
device itself to open its configuration dialog box.

• In Username and Password text boxes, type the credentials that will be used to access the
device and click Add. An API key will be automatically generated.

Expedition Security Policy Optimization 11

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Go to CONTENTS tab and click the arrow next to retrieve contents. Select Running
Configuration.

• Go to PANORAMA DEVICES and click Retrieve Connected Devices. After all devices
managed by Panorama have been retrieved, click Save. The managed devices should now
appear in the list of imported devices along with Panorama.

Expedition Security Policy Optimization 12

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Double click the firewall you want to analyze and go to the M. LEARNING tab. In the logs PATH
text box, type the path where the logs can be found (in our case it will be /PALogs/). Click Search
Files. The list of available files should appear in the bottom pane.

• Click Process Files. After a few seconds, a link should appear at the bottom of the dialog box. By
clicking it, you’ll open a new browser tab showing the status of the processing by the Spark
engine. Keep refreshing the tab until it displays an error which indicates that processing is
complete. Click Save.
Step 8 – Create a new Expedition project.
• From Expedition GUI, go to PROJECTS and click the + sign at the right to create a new project.
In the dialog box, type a Name for the project and select Greenfield ML as the purpose of the
project. Click Create Project.

Expedition Security Policy Optimization 13

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • The new project should appear in the list of projects. Select the project in the list and click
Settings at the right. Click the DEVICES tab and select the devices related to your project. Click
the arrow pointing to the right, Add Access. Click Save.

• Double-click your project, then click on the IMPORT tab. Your devices should appear in the
DEVICES tab in the bottom pane. Select the Panorama device and click Import Device button at
the bottom right. Device import starts, and after a few moments, the Global Summary page
under Dashboard tab should appear.

Expedition Security Policy Optimization 14

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Go to PLUGINS > PAN-OS CONNECTORS and click the + sign at the right. Give the connector a
Name. In the Select Device drop-down box, select the Panorama device. Choose the
appropriate Device Group in the next drop-down box. In the Filter by Devices in the Device
Group pane, select the firewall you want to analyse logs for. In the Period of Time to Analyze
pan, choose the time span.

Step 9 – Learn and create the new rules in Expedition.

As we want to identify the application our users are using on the public internet, we will enable machine
learning on the rule that allows access on ports 80 and 443, and let Expedition analyze the logs and
come up with a suggested list of rules.
• Go to Policies > Security tab, then select your device group in the bottom right drop-down box.
Right click the rule you want to learn traffic from then go to Machine Learning > Monitor >
Selection. An ML Enabled tag should now appear next to the name of the policy.

Expedition Security Policy Optimization 15

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • In Policies > Security tab, click the Discovery button at the bottom of the page. In the dialog box
that opens up, make sure you’re under ANALYSIS RESULT tab. Select the time frame for the
analysis if it’s different from what has been configured under the log connector, click Analyze
Data. After a few seconds, a link at the bottom of the page should appear. Click on it to follow the
progress of the analysis. Keep refreshing the new tab until an error message is displayed, which
indicates that processing is complete.

• After the processing is complete, a list of learned security rules should appear in the Learning
Results pane. Before importing the rules into project, you might want to review them. By clicking
on the arrow next to each column, you have a list of options that allow you display the rules in the
way that makes it easier for you to review. In our case, we want to group the rules by application
to review them. Additionally, the rules can be exported to an excel file and shared for additional
review.

Expedition Security Policy Optimization 16

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Click IMPORT INTO PROJECT on the right pane. In the Apply To options, select All Rules. In
the Objects, deselect Address, as we want our rules to apply to all internet users by having the
source address set to any. Click Import. The status in the bottom pane should change to
pending, then Done! Click Close.

Expedition Security Policy Optimization 17

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Back to Policies > Security tab, new rules should now appear below our EXPLICIT-DENY rule.
Each of the rules allows a specific application. If there’s an application that you don’t want to
allow, you can still delete the corresponding rule by selecting it and clicking the – sign at the top
right corner. We are going to merge all these rules into a single one and move it before the rule
on which we enabled machine learning. Select all the newly added rules and right-click on one of
them. Click on Rules Action > Merge. Click OK on the dialog box that confirms the merger.

Expedition Security Policy Optimization 18

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • Select the resulting rule and click the Move button at the bottom. Make sure that TOP is selected
in the Move To drop-down box and click Move.

• Our rulebase should now look like the below.

Expedition Security Policy Optimization 19

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Step 10 – Export the newly created rules to the firewall.
In this last step, we will use Expedition to create these rules on Panorama and push them to the firewall.
• Go to EXPORT > API Output Manager and select the SubAtomic radio button. Click Generate
API Requests button at the bottom. After a few seconds, a list of API calls should be displayed.

• In the list of API commands, locate the one specific to the new security rule we want to add (ID 15
in our case) and double-click it. Make sure that the selected device is the Panorama device in the
bottom drop-down box and click Send API Call. After a few moments, a “command succeeded”
message should appear.

Expedition Security Policy Optimization 20

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 • From the Panorama GUI, go to Device Groups > Policies and make sure the new policy is
there. Click Commit > Commit and Push at the top right.

Step 11 – Repeat the process.

With our syslog configuration, one log file is created for each day. After the initial learning and rules
creation process, our initial rule will probably keep matching traffic and generating logs, which will be
available for Expedition to process. The more iterations of the process, the less amount of traffic will hit
the initial rule. At some point, you’ll have to decide whether you’re happy with the rules that have been
created regarding their coverage of the requirement, and stop the learning process.
Note: When repeating the process, you’ll need to create a new project in Expedition for each new
iteration.

Expedition Security Policy Optimization 21

via Machine Learning Proprietary and Confidential ©2019 Palo Alto Networks, Inc.