Functional Design Specification OMC Cloud Parser

- Oracle AVDF Alert Event

Abstract: This is the functional design specification of Oracle Audit Vault and
Database Firewall Alert Event Security Parser

MRD: - May, 19 2017

Feature(s): -

Author: sshahed

Version: 1.0(Draft)

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 1 of 13

Template version: 1.0
Revision History
Date Who Description of What Was Changed
19-May-2017 sshahed Initial Version

Reviewers
Date Who Team Comments

Approvers
Date Who Title Comments

Delivery Checklist
Item Date Comments
Function Design Document 19-May-
2017
OSEF Mapping 19-May- Attached to confluence page
2017
Sample Logs 19-May- Attached to confluence page
2017
Base parser (logtypes) 19-May- Attached to confluence page
2017
Extended parsing and tagging 19-May- Attached to confluence page
(sources,efd,etc) 2017
Oracle JSON validation TBD
Oracle production signoff TBD

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 2 of 13

Template version: 1.0
Table of Contents
Revision History........................................................................................................................................................ 2
Reviewers................................................................................................................................................................. 2
Approvers................................................................................................................................................................. 2
Delivery Checklist..................................................................................................................................................... 2
Table of Contents...................................................................................................................................................... 3
1 Background....................................................................................................................................................... 4
1.1 Terminology................................................................................................................................................ 4
1.2 References:................................................................................................................................................ 5
1.3 Setup Instructions:..................................................................................................................................... 5
1.4 Platforms Support...................................................................................................................................... 5
1.5 Version Supported...................................................................................................................................... 5
2 Design Contacts................................................................................................................................................ 5
3 Use Cases......................................................................................................................................................... 6
4 LA Parser Considerations.................................................................................................................................. 6
4.1 Entity type:................................................................................................................................................. 6
4.2 Source Type............................................................................................................................................... 6
4.3 Base parser (logtypes & sources).............................................................................................................. 6
4.4 Extended parsing and tagging sources:..................................................................................................... 6
4.5 Curl Command:.......................................................................................................................................... 6
5 Analysis............................................................................................................................................................. 7
5.1 Details on AVDF msg field payload (in Syslog)........................................................................................ 7
5.2 Details on AVSYS.EVENT_LOG Table (in AVS DB)...................................................................................8
5.3 Details on AVSYS.ALERT_STORE Table (in AVS DB)...............................................................................9
6 Design Discussion............................................................................................................................................. 9
6.1 Key SEF Field Mapping.............................................................................................................................. 9
6.2 SEF Mappings for AVDF Alert Events (in Syslog).......................................................................................9
6.3 SEF Mappings for AVDF Alert Events (in AVS Database Tables)...............................................................9
7 Example JSON................................................................................................................................................ 10
7.1 Syslog log file........................................................................................................................................... 10
8 Issues.............................................................................................................................................................. 12
Terms and Acronyms.............................................................................................................................................. 13

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 3 of 13

Template version: 1.0
1

2 Background
This document is to track design and implementation discussions surrounding SEF normalization of Oracle Audit
Vault and Database Firewall (AVDF) Syslog Messages. It will not discuss how to setup and configure an AVDF
system instead will focus on how to parser and do SEF normalization and tagging of Audit Vaults file based log
file.

Once an AVDF system is running, the Audit Vault Server (AVS) can be configured to send the AVS audit logs onto
another host machines syslog file. This syslog file from that host which is outside of AVDF system can then be
consumed and processed by OMCs Log Analytics and eventually in Security Analytics.

Oracle AVDF can monitor following databases:

Oracle
MySQL
Microsoft
IBM
Sybase

It also can monitor following non-database targets:

OS
Directory Services
File System
Custom Audit Logs

Initial implementation for OMC Security Monitoring and Analytics (SMA) is focused on Audit Vault logs using Unix
Syslog files. In near future, we should also be able to access same audit log data from AVS database table(s).

2.1 Terminology
Oracle AVDF includes the Audit Vault Server, the Database Firewall, and the Audit Vault Agent.

The process flow for the Oracle AVDF components is as follows:

1. For each secured target, the Audit Vault Agent has been deployed, and/or the Database Firewall has
been placed in the network and configured to protect that target.

If the agent has been deployed, Oracle AVDF is configured to collect the appropriate audit trail from the
secured target. If the Database Firewall is protecting the target, a firewall policy has been applied for that
target.

You can configure multiple secured targets from different database product families, as well as non-
database products, using the same Audit Vault Server.

2. The Audit Vault Agent retrieves the audit data from secured targets and sends this data to the Audit Vault
Server.

The Database Firewall monitors SQL traffic to database secured targets and sends that data to the Audit
Vault Server. The firewall can be configured to monitor and raise alerts only, or to block SQL traffic and
optionally substitute statements according to a policy.

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 4 of 13

Template version: 1.0
 3. The Audit Vault Server collects and stores data from the Audit Vault Agent and Database Firewall in its
internal data warehouse.

The data warehouse organizes this data into a set of internal dimension tables. The Audit Vault Server
stores other information as well, for both the auditor and the administrator.

4. Once the audit data is in the data warehouse dimension tables, an auditor can generate and customize
reports, as well as configure email notifications.

Any settings that you create, such as security settings, are contained in this server.

Oracle AVDF includes the Audit Vault Server, the Database Firewall, and the Audit Vault Agent.

2.2 References:
1. https://docs.oracle.com/cd/E37100_01/doc.121/e27777/intro.htm
2. https://docs.oracle.com/cd/E37100_01/doc.121/e27777/schema.htm
3. http://docs.oracle.com/cd/E69292_01/index.html
4. https://docs.oracle.com/cd/E37100_01/doc.121/e27777/schema.htm#SIGAU40278

2.3 Setup Instructions:

Provide details on how to setup and configure product to generate the required logs and related
messaged within the log.

1. Oracle AVDF needs to create all required security policies that are relevant for OMC SMA
2. Oracle AVS server needs to forward Audit logs to Syslog in a host machine that can run an OMC LA
agent.
3. OMC 1.18+ that includes Oracle AVDF LA parser.

2.4 Platforms Support

Linux/Unix
o Other (please specify)

2.5 Version Supported

Describe the product versions that are supported by the parser. Any potential caveats.

Oracle AVDF 12c (Database Security 12c Hands-on Workshop V 9.3)

3 Design Contacts
OSEF team lead developer: sumon.shahed@oracle.com
Backup developer:

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 5 of 13

Template version: 1.0
4 Use Cases
Define various security use case that this parser data can utilized by the SMA team.

5 LA Parser Considerations
Outline the components of the parser.

5.1 Entity type:

No new LA Green Field Type needed
Extracted from: db\xml\loganalytics\sources\ em_db_avdfalert_syslog_source.xmlp
o <TargetTypes>
<TargetType>omc_host_linux</TargetType>
</TargetTypes>
entityType=omc_host_linux

5.2 Source Type

Extracted from: db\xml\loganalytics\sources\ em_db_avdfalert_syslog_source.xmlp
o <DisplayName nlsid="logan_nlsid"> AVDFAlert Linux Syslog </DisplayName>
o logSourceName=AVDFAlert%20Linux%20W3C%20Syslog

Currently using /var/log/messages* as file name

5.3 Base parser (logtypes & sources)

host\xml\loganalytics\logtypes\em_host_syslog_logtype.xmlp
db\xml\loganalytics\sources\ em_db_avdfalert_syslog_source.xmlp

5.4 Extended parsing and tagging sources:

Extended Fields
o em_db_avdfalert_extfields.xmlp
tags
o em_db_avdfalert_tags.xmlp

5.5 Curl Command:

Generated by oracle for remote upload of this parsers log file using entity and source type listed above.

curl -v -X POST -k -u "${TENANT_ID}.emcsadmin:Welcome1!" -H "X-USER-IDENTITY-

DOMAIN-NAME:${TENANT_ID}" --form 'data=@em_db_avdfalert_syslog' "https://$
{OHS_HOST}/serviceapi/logan.uploads?
uploadName={ANY_NAME}&logSourceName=AVDFAlert%20Linux%20W3C
%20Syslog&entityName={OHS_HOST}&entityType=omc_host_linux"

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 6 of 13

Template version: 1.0
 cat em_db_avdfalert_syslog.log
Feb 5 03:39:01 dbsec12c rsyslogd: [origin software="rsyslogd"
swVersion="5.8.10" x-pid="1768" x-info="http://www.rsyslog.com"] rsyslogd was
HUPed
Feb 6 16:53:56 avs kernel: imklog 5.8.10, log source = /proc/kmsg started.
Feb 6 16:53:56 avs rsyslogd: [origin software="rsyslogd" swVersion="5.8.10"
x-pid="22561" x-info="http://www.rsyslog.com"] start
Feb 6 16:53:56 avs rsyslogd: WARNING: rsyslogd is running in compatibility
mode. Automatically generated config directives may interfer with your
rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the
first rsyslogd option.
Feb 6 16:54:23 avs run1: com.oracle.dbfw.run1 INFO - CLI Executed

6 Analysis
OMC LAs base Syslog file parser processes each log entry and creates following fields:
TIMESTAMP
srvrhostname
service
ospid
msg

Each log entry can contain multiple lines.

In case of AVDF log as shown in Appendix->Sample Data, base parser will extract:
TIMESTAMP (e.g. Feb 6 16:55:27)
srvrhostname (e.g. avs)
service (e.g. logger)
msg (e.g. [AVDFAlert@111 (EVENT(1/1)=(AN= )

An Extended field parser was created to further parse the payload in LA base fields named msg to required
fields needed for SMA. Please note the order of relative positions of these fields in AVDF log is fixed.

The SEF tagging will be done by avdfalert_tags using all the fields created by Extended field parser.

6.1 Details on AVDF msg field payload (in Syslog)

Following table describes the semantic meaning of various fields that are present in AVDF msg payload.

AVDF Semantic Meaning Possible Values Comment

Audit Log
Field
Name
(raw log)
Alert Name (Set in "Failed Login Oracle", "Super User Login",
AN Oracle AVDF policy) "createUser", "Database Firewall Alert"
Alert Time (Time when
AT the event was recorded 2017-02-06T21:56:06.630803-05:00
in Oracle AVDF
repository.)
"Critical", "Warning"
ASE Alert SEverity

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 7 of 13

Template version: 1.0
AST Alert STatus "New"
N/A
AD Alert Description (An
alert evaluated at a
Database Firewall,
based on a Firewall
Policy.)
Url to the AVDF
URL application for the https://192.168.56.200/console/f?p=7700:33:::NO::P33_ALERT_ID:2245
details of the Event.
Alert ID
AID 2245
Secure Target Name
STN "DBSecOS", "db12sjis"
Secure Target Type
STT "Linux", "Oracle Database"
Event Name
EN "SUDO_SUDO", "SELECT"
Event Time
ET 2/6/2017 11:55:40 AM UTC
Event Status
ES "SUCCESS","FAILURE", or "UNKNOWN"
Command Class
CC "EXECUTE", "SELECT", "DELETE"
Operating System User
OSUN Name "oracle"
User Name
UN "oracle"
Client Host Name
CHN (Name of client host N/A
where the user started
the action)
Client IP (of CHN)
CIP N/A
Client Program (Client "/usr/bin/sudo", "JDBC Thin Client"
CP program where the
event occurred)
Target OBJect
TOBJ "/usr/bin/sudo",
"ALL_UNIFIED_AUDIT_ACTIONS"
Target TYPE
TTYPE "PROGRAM", "TABLE"
Target OWNer
TOWN "root", "SYS", "HR"
Audit Trail Type
ATT "TABLE" or "DIRECTORY"
Audit Trail Location
ATL "/var/log/audit/audit.log",
"UNIFIED_AUDIT_TRAIL"
ACTION taken (Action
ACTION taken for the event) "pass", "warn", or "block"
Policy Name(Name of
PN policy file that the N/A

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 8 of 13

Template version: 1.0
 Database Firewall used
when it detected the
event)
Threat Severity
TS (Severity of the threat undefined, insignificant, minor, moderate,
that the Database major, or catastrophic.
Firewall detected)
Cluster Type
CT 5
Cluster ID
CID "355210392"
Grammar Version
GV "8036"
Log Cause (Cause of
LC the event, as recorded undefined, exception, cluster, novelty,
in the log) unseen, invalidsql, waf, login, or logout
Error Code
EC N/A

6.2 Details on AVSYS.EVENT_LOG Table (in AVS DB)

Visit https://docs.oracle.com/cd/E37100_01/doc.121/e27777/schema.htm#CFHEAGFA

6.3 Details on AVSYS.ALERT_STORE Table (in AVS DB)

Visit https://docs.oracle.com/cd/E37100_01/doc.121/e27777/schema.htm#CFHEJCIJ

7 Design Discussion

This section describes required SEF mapping for useful AVDF fields for OMC SMA.

7.1 Key SEF Field Mapping

sefLogFormat=em_host_syslog_avdfalert
sefObserverEPManufacturer=Oracle
sefObserverEPProduct=Oracle Audit Vault and Database Firewall
sefObserverEPType=AVDF
mseccategory(category)=alert

7.2 SEF Mappings for AVDF Alert Events (in Syslog)

AVDF LA Field Name SEF Field Name Comment.

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 9 of 13

Template version: 1.0
Audit Log
Field
Name
(raw log)
AN sefAction
AT sefEndEventTime
ASE sevlvl
URL detailloc detailloc should be detailInfoUrl
STN sefSourceEPName
STT sefSourceEPType
EN eventid eventid should be eventname (does not
exist in LA as of now)
ET sefStartEventTime
ES status
CC sefCommand
UN sefSourceEPAccountName
CHN sefActorEPName
CIP sefActorEPNwAddress
TOBJ eventtarget
TTYPE eventtargettype
TS sefSourceEPAccountSummaryRisk

7.3 SEF Mappings for AVDF Alert Events (in AVS Database Tables)
In progress: needs to be implemented and tested first. Need access to AVS DB populated with useful
data.

AVSYS.EVENT_LOG Table
Column Name LA Field Name SEF Field Name Comment.
ACTION TBD
AV_TIME sefEndEventTime
SECURED_TTARGET_NAME sefSourceEPName
SECURED_TARGET_TYPE sefSourceEPType
EVENT_NAME eventid
EVENT_TIME sefStartEventTime
EVENT_STATUS status
COMMAND_CLASS sefCommand
USER_NAME sefSourceEPAccountName
CLIENT_HOST_NAME sefActorEPName
CLIENT_IP sefActorEPNwAddress
TARGET_OBJECT eventtarget
TARGET_TYPE eventtargettype
THREAT_SEVERITY sefSourceEPAccountSummaryRisk

AVSYS.ATERT_STORE Table
Field Name LA Field Name SEF Field Name Comment.
ALERT_NAME sefAction TBD
ALERT_SEVERITY sevlvl

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 10 of 13

Template version: 1.0
No AVSYS TABLE was found for following fields that are present in AVDF Event in Syslog:
URL

8 Example JSON

8.1 Syslog log file

JSON for [Appendix].Sample Data LINE #2

{
"id":"b6a99afb511a3dd2b2e5d858058c28020a7A0",
"mguid":"b6a99afb511a3dd2b2e5d858058c28020a7",
"mlogent":"Joydip_EMCSAS-3839.log",
"mtgtguid":"AA84B1E5C965AB239FE2095158EEFBA2",
"muploadid":-462246114838981815,
"mtgttype":"omc_host_linux",
"mtgt":"AVDFJoydipData",
"msrcid":-1988864882359077022,
"time":"2017-02-06T16:56:06Z",
"mparserid":-1353330754,
"mdstime":"2017-04-26T20:00:31.796Z",
"mprtime":"2017-04-26T20:00:35.264Z",
"sefLogFormat":"em_host_syslog_avdfalert",
"detailloc":"https://192.168.56.200/console/f?p=7700:33:::NO::P33_ALERT_ID:2244",
"sefCommand":"EXECUTE",
"sefSourceEPName":"DBSecOS",
"status":"SUCCESS",
"eventtargettype":"PROGRAM",
"mseccategory":"alert",
"sefSourceEPAccountName":"oracle",
"mbody":"Feb 6 16:56:06 avs logger: [AVDFAlert@111 (EVENT(1/1)=(AN=\"Super User
Login\" AT=\"2017-02-06T21:56:06.619261-05:00\" ASE=\"Critical\" AST=\"New\" AD=\" \"
URL=\"https://192.168.56.200/console/f?p=7700:33:::NO::P33_ALERT_ID:2244\" AID=\"2244\"
STN=\"DBSecOS\" STT=\"Linux\" EN=\"SUDO_SUDO\" ET=\"2/6/2017 11:55:33 AM UTC\" ES=\"SUCCESS\"
CC=\"EXECUTE\" OSUN=\"oracle\" UN=\"oracle\" CHN=\"\" CIP=\"\" CP=\"/usr/bin/sudo\"
TOBJ=\"/usr/bin/sudo\" TTYPE=\"PROGRAM\" TOWN=\"root\" ATT=\"DIRECTORY\"
ATL=\"/var/log/audit/audit.log\" EP=\"\" EPM=\"\" ACTION=\"\" PN=\"\" TS=\"\" CT=\"\"
CID=\"\" GV=\"\" LC=\"\" EC=\"\"))]",
"msg":"[AVDFAlert@111 (EVENT(1/1)=(AN=\"Super User Login\" AT=\"2017-02-
06T21:56:06.619261-05:00\" ASE=\"Critical\" AST=\"New\" AD=\" \"
URL=\"https://192.168.56.200/console/f?p=7700:33:::NO::P33_ALERT_ID:2244\" AID=\"2244\"
STN=\"DBSecOS\" STT=\"Linux\" EN=\"SUDO_SUDO\" ET=\"2/6/2017 11:55:33 AM UTC\" ES=\"SUCCESS\"
CC=\"EXECUTE\" OSUN=\"oracle\" UN=\"oracle\" CHN=\"\" CIP=\"\" CP=\"/usr/bin/sudo\"
TOBJ=\"/usr/bin/sudo\" TTYPE=\"PROGRAM\" TOWN=\"root\" ATT=\"DIRECTORY\"
ATL=\"/var/log/audit/audit.log\" EP=\"\" EPM=\"\" ACTION=\"\" PN=\"\" TS=\"\" CT=\"\"
CID=\"\" GV=\"\" LC=\"\" EC=\"\"))]",
"eventid":"SUDO_SUDO",
"srvrhostname":"avs",
"sefObserverEPProduct":"Oracle Audit Vault and Database Firewall",
"eventtarget":"/usr/bin/sudo",
"service":"logger",
"sefObserverEPType":"AVDF",
"sevlvl":"Critical",
"sefSourceEPType":"Host",
"sefAction":"Super User Login",
"sefObserverEPManufacturer":"Oracle",
"norm_hashid":5897917602427668891,
"sefRecordType":"base",
"sefRecordGuid":"5ff63992-a773-3a1b-904f-7318620ef76f",

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 11 of 13

Template version: 1.0
 "msmaannotated":1,
"sefDestinationEPType":"omc_host_linux",
"sefCollectorEPType":"omc_host_linux",
"sefEndEventTime":"2017-02-06T16:56:06Z",
"sefObserverEPGuid":"42BFFAB52A5E2F621EF5BC1538883B73",
"sefActorEPAccountName":"omc_unknown_account",
"sefCollectorEPName":"AVDFJoydipData",
"sefResult":"success",
"sefAddlAttrs":["eventTargetType:PROGRAM",
"mdstime:2017-04-26T20:00:31.796Z"],
"sefEnrichmentTime":"2017-04-26T20:00:45.828Z",
"sefCategory":"alert",
"sefDestinationEPName":"AVDFJoydipData",
"sefStartEventTime":"2017-02-06T16:56:06Z",
"sefActorEPGuid":"F6D4218F9770D91A1658A6ADFF2EFB7B",
"sefSeverityNum":0,
"sefActorEPEffectiveAccountName":"omc_unknown_account",
"sefCollectorEPGuid":"AA84B1E5C965AB239FE2095158EEFBA2",
"sefActorUserPrimaryOrg":"0",
"sefContributorCount":1,
"sefActorUserName":"User_Internal_omc_unknown_account",
"sefDestinationEPGuid":"0D57028216DC4B020B2C8CBFE20B577F",
"sefDestinationEPAccountName":"omc_unknown_account",
"sefSourceUserName":"User_Internal_oracle",
"sefResultCode":"SUCCESS",
"sefUnitsTransferred":0,
"sefSourceEPGuid":"09838C9E982F5F62C16863D855F77F2F",
"_version_":1565772319079530496}

9 Issues
Need real AVDF Audit Logs to cover above mentioned use cases

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 12 of 13

Template version: 1.0
Terms and Acronyms
Term Meaning
MRD Material Required Date

Updated:12/10/2017 2:34:48 PM Proprietary and Confidential to Oracle Corporation Page 13 of 13

Template version: 1.0