1040 Bennacer Complete

Managing an Arcsight
Express 3.0 appliance

Samir Bennacer & Nathan Tisdale
ArcSight Advanced Support Engineers
September 19, 2012
Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Agenda
Overview
Application health
Events in
Events out
Managing event archives

Hardware health
Streamlining the RMA process
Managing ArcSight Express 3.0:

Overview of appliance
Data to information
ESM 101: Concepts for ESM + CORR-Engine provides a full overview of event flow.
Connectors
Smart
Connectors
collect &
normalize
events form
environment
Network model
Details
regarding
networks,
assets, zones
etc.
Use case
Customizing
rules, data
monitors, lists, to
fit your business
needs
Report
Dashboards
Traditional
reports
ArcSight Solution
Packages
Hierarchy
Scale with
additional
Appliances to
accomodatee
volume and
reporting needs
Connecting to Express 3.0

ArcSight Console
Interactive Web Base Monitoring
Traditional Console also available
Management Console
Manage Users accounts, storage, connectors, notifications and license
HP Integrated Lights-Out (iLO)

Secure remote management regardless of server status or location.
See KM1272897 for details regarding setup of iLO.
ArcSight Express 3.0 Management Console

Event Storage is for daily events that are younger than the retention period. Archives
are used to preserve events offline, beyond the retention period.
Storage management

You can add and delete users and groups, and perform other user management
functions
Administration

Dashboards in the Management Console appear as layouts of dashboard data using a browserbased runtime environment. You can see all the dashboards that appear in the ESM Console and
you can rearrange the layouts and save them
Dashboards
AE 3.0 Advanced Administration

https://ipadress:8443/arcsight/web/manage.jsp
ArcSight Express 3.0 Process Management

Single script to manage all ArcSight services
Control process dependence and startup sequence
Restart failed service
Unified control of all ArcSight services with /sbin/service arcsight_services

arcsight_services help
arcsight_services [start | stop | status | ] [all | logger | manager | mysqld | ]
10
ArcSight Express 3.0 Directory Layout

/opt partition, a separate partition, 64bit XFS, total space 1.5TB
All ArcSight software & data under single directory: /opt/arcsight
This directory is owned by user arcsight.
All arcsight operations should be run as arcsight, not root
ESM manager: /opt/arcsight/manager
ESM web: /opt/arcsight/web
Appliance process management: /opt/arcsight/services
Logger: /opt/arcsight/logger
MySQL: /opt/arcsight/logger/data/mysql
Event archive directory: /opt/arcsight/logger/data/archives
1 directory per day
Event storage directory: /opt/arcsight/logger/data/logger
1GB per file
11
ArcSight Express 3.0 Storage Allocation
Total storage: 1.6TB

Root partition: 100GB
Event Storage: 919GB
None event storage(InnoDB): 200GB
Event archival storage: 200GB

The rest: ESM binary/log files/Posgres DB/MySQL temporary sort area
12
ArcSight Express 3.0 Storage Engines

The CORR-Engine relies on MySQLs pluggable storage engine architecture
Allows for different types of data handling
ArcSights high performance event storage and retrieval
InnoDB Built-in transactional support, allowing updates and deletes

Multiversion concurrency control (same as Oracle)
Used for ESM resources(rules, channels, ) & trend data, active / session list data, annotations
MySQL seamlessly handles the joins (e.g.: events and cases, actors)
Patent-pending technology superstore (single database with Row and Column
store)
13
CORR-Engine: Overview
CORR-Engine
C
o
C
O
E m
n m
g
ESM
R
i L
Manager
R
-
n a
e y
e
r
14
Events
Logger
Event
ArcSight
Event
Store
Server
Store
Events
MySQL
Resources
and data
Logger
Storage
Engine
InnoDB
Storage
Engine
ArcSight Express 3.0 Events

All event fields are indexed
No penalty for accessing any event fields
Compare with previous generation having only 16 indexed arc_event fields
Remove side tables completely

No more need to manage side table cache sizes
No more descriptor side table flooding issues
Fold arc_event_ [agent, category, device, label, geo_location, correlation] into arc_event
Better query performance due to elimination of event join with side tables
15
ArcSight Express 3.0 DB Management

No partition compressor needed
No partition table needed
System built-in optimizer
No database statistic job
No query tuner needed
No user level query hint required(table/index/different join type etc)
No event time correction(clamping) required
16
Managing ArcSight Express 3.0

Application health
Troubleshooting feature behavioral issues

Process remains same ESM troubleshooting of issues with same nature
Typical Steps
Identify
Event Flow
System/Services Offline
Error
Unexpected Behavior
Wish it did
Collect Data
Model
Version
Logs
Engage support
Support Center
18
Command line application check

/etc/init.d/arcsight_services status | start | stop
All services are running
Manager is not running
web service is available

manager service is available
execprocsvc service is available
logger service is available
mysql service is available
postgresql service is available
web service is mixed_statuses

manager service is unavailable
execprocsvc service is unavailable
logger service is unavailable
mysql service is available
postgresql service is unavailable
19
Logs & more

Need to investigate application behavior?
Writing events (caching?)
Reading events (channels/reports)
Behavior (startup / content)
Manager logs
Manager logs
Manager logs
Thread dumps
Database logs
System tables
DB sessions
Manage.Jsp
Database logs
Collect and upload logs to Service Center
20
Collecting application logs

SENDLOGS
Send Logs wizard remembers most of the choices you make when you run it for the first time
Local logs only
Time Range
Including other components
Connectors
CORR-Engine
Diagnostics (runsql, session-waits, threaddumps)
Time Range
Sanitize
Incident number (for naming file)
21
Collecting database logs

ARCDT
This diagnostic tool can be used as part of sendlogs or run separately as needed.
runsql
Create sql file [ e.g. select count(*) from arc_resource where resource_type=1; ]
/opt/arcsight/manager/bin/arcsight arcdt runsql f usercount.sql
session-waits
/opt/arcsight/manager/bin/arcsight arcdt session-waits
thread-dumps
/opt/arcsight/manager/bin/arcsight arcdt thread-dumps
22
Summary of Express 3.0 changes
Single script to Manage all ArcSight services

CORR-Engine improves read and write performance
Compression of events when persisted
Less need to use Trends to boost reporting performance
Elimination of descriptor Side Tables boosts performance

Web Management Interface
Collect logs with sendlogs
Archival without Partition Management
23

Managing event archives
Event retention
Max event storage size: 919GB
MRT(manager-receipt-time) based data retention(not end time)
Oldest events will be overwrite first(FIFO)
Events pruning based on the age of events
Email notifications can be configured to alert people

During First Boot Wizard
Via management console (web UI)
25
Configuring event retention
26
Storage layout
Event storage layout
1-Jul
2-Jul
3-Jul
4-Jul
5-Jul
6-Jul
27
The events portion of the CORR-Engine storage

management system consists of two major parts:
The active Retention period
Defined by max age or space. This defines the on-line events.
Archives
Data files containing events of one day, which have been copied to the archive location, with two
additional files containing metadata related to these data files.
28
Event flow
29
Archiving process
A subdirectory with the format YYYYMMDD is created for each day to store the archive. If the directory is
already present, then it is removed first.
Two metadata files, an XML and CSV, for the archival information are created
These files are compressed and copied to the archiving location.
Finally, the data files containing the events are copied to the archive location.
Note : The archiving is continual and is copying the events files offline as opposed
to what used to be in ESM where the same partition was taken offline. Does not
affect events in online retention
30
Archiving modes
There are two modes of archiving:
Scheduled - Runs on a daily basis, archiving the events from the day before.
Manual (user-driven)
We recommend to use scheduled mode and use the manual mode for retrying an
unsuccessful scheduled one.
31
Where the archives reside
32
Changing the size and the location of the archive folder is is not supported in AE 3.0
If there is no more space the archiving will fail.
We can monitor the space used in the management-ui
Mounting external drive to the archive directory is not supported.
Configuring archive schedule

Time to start the archive operation for the current day's events as well as any days
manually marked for "retry":
33
Manual archive (user-driven)

Manual mode for retrying an unsuccessful scheduled one.
34
Archive jobs
35
...Still in active storage

Each archive will represent one day's worth of events
The number of archives in this list will be the number of days that fit in the configured retention policy
constrained by both a time-dimension and a space-dimension e.g. a retention policy of 30 days will have up
to 30 items in the list .. less if there is not enough space.
For each item(archive) in the list there will be identifying information: Date, Archive ID
The different states that these archives can be in are Pending, In-progress, Archived, Not-Archived.
36
Archives
37
2. No longer in active storage

Initially will be in the Deactivated state thus events are not accessible.
Can be in one of the following states Deactivated, Activating, and Activated
38
Auditing and logs

Device event class
ID Audit event description
archive:100
Archive created
archive:101
Archive deleted
archive:102
Event archive settings updated
archive:103
Event archive disk space used
archive:110
Archive activated
archive:111
Archive activation cancelled
archive:112
Archive activation failed
archive:120
Archive operation succeeded
archive:121
Archive operation cancelled
archive:122
Archive operation failed
archive:130
Archive deactivated
archive:131
Archive deactivation cancelled
archive:132
Archive deactivation failed
archive:140
Archive scheduled
archive:141
Archive schedule cancelled
archive:142
Archive schedule failed
39
We can see the Archive auditing in the Server.std.log and server.log

{fileType=Archive, cat=/Monitor/Archive/Create, cs2=20110913, severity=1, msg=Created archive
20110913, fileId=0504403158265495552, deviceEventClassId=archive:100, start=1315952153371,
name=Archive created, rt=1315952153371, fname=20110913, cs2Label=Archive Name,
end=1315952153371, fpath=/opt/arcsight/logger/data/archives/20110913}
INFO | jvm 1 | 2011/09/13 15:22:35 | {fileType=Archive,
cat=/Monitor/Archive/Configuration/Scheduling/Success, cs2=20110913, severity=1, msg=Successful
scheduling of archive 20110913, fileId=0504403158265495552, deviceEventClassId=archive:140,
start=1315952153648, name=Archive scheduled, rt=1315952153648, fname=20110913, cs2Label=Archive
Name, end=1315952153648, fpath=/opt/arcsight/logger/data/archives/20110913
40
Service:EventArchiveManager
Look for Service:EventArchiveManager in the server.status.log
[2011-10-03 03:55:22,456] Service:EventArchiveManager
[2011-10-03 03:55:22,456] ObjectName:Arcsight:service=EventArchiveManager
[2011-10-03 03:55:22,539]
ArchiveEnabled="true"
[2011-10-03 03:55:22,623]
ArchiveSchedule4Display="Wed Dec 31 01:00:00 PST 1969"
[2011-10-03 03:55:22,623]
ConfiguredDaysInRetentionPolicy="0"
[2011-10-03 03:55:22,666]
DiskspaceFree="210453397504"
[2011-10-03 03:55:22,708]
LocationOnDisk="/opt/arcsight/logger/data/archives"
[2011-10-03 03:55:23,559]
OfflineArchives="[
 archiveDate=[Fri Sep 30 00:00:00 PDT 2011 --> 1317366000000millis]

 archiveID=[0504403158265495564]
 STATE=[ACTIVE] ; online=[false] ; eventCount=[0] ; diskspaceConsumed=[0]
 startDateOfEvents=[Fri Sep 30 00:00:00 PDT 2011 --> 1317366000000millis]
 endDateOfEvents=[Fri Sep 30 23:59:59 PDT 2011 --> 1317452399999millis]
 AdditionalInfo=[null]
 dateActivated=[Sun Oct 02 08:17:30 PDT 2011 --> 1317568650860millis]
 dateDeactivated=[Sun Oct 02 03:35:24 PDT 2011 --> 1317551724859millis]
 dateArchiveStarted=[Wed Jan 01 00:00:00 PST 9500 --> -361932940800000millis]
 dateArchiveCompleted=[Wed Jan 01 00:00:00 PST 9500 --> -361932940800000millis]
 scheduledStartDateOfArchiveOperation=[Sun Oct 02 08:17:30 PDT 2011 --> 1317568650833millis], ]
41
How to restore archive backups to a new system

after a system failure?
Always Stop logger service /sbin/service arcsight_services stop logger
To test how many archives will be restored and see if any are unreadable.
Run /opt/arcsight/logger/current/arcsight/logger/bin/arcsight restorearchives -t
To clear any existing events from the system. and then register all the backup archives you placed in
/opt/arcsight/logger/data/archives
Run /opt/arcsight/logger/current/arcsight/logger/bin/arcsight restorearchives C
To register all the backup archives you placed in /opt/arcsight/logger/data/archives.
Run /opt/arcsight/logger/current/arcsight/logger/bin/arcsight restorearchives
-Start Logger Service /sbin/service arcsight_services start logger
42

Hardware health
Appliance fails to boot

System or OS level failure
Following these steps and contact support for analysis.
Connect to the appliance using iLO, KVM switch or using keyboard/monitor physically connected to the
back
Get screen shot or photo showing the boot-up failure
e.g. errors, warnings, failures to start services, etc.)
Errors should indicate if the problem is software or hardware
In case of File System inconsistency the system prompts to run fsck (file system check) to fix it.
44
Refer to KM1272371 for more details
Check /var/log/messages and dmesg in single user mode for errors

Retrieve logs
Diagnosing tools
Commands to validate hardware failure:
HP Appliances (RAID Controller and disks)
hpacucli ctrl all diag file=hp-raid-diag-output.zip ris=off xml=off zip=on

hpacucli ctrl all show config detail
hpacucli help
HP iLO webUI
ipmitool sel
ipmitool sensor
dmesg
cat /var/log/messages
45
Summary of resources for diagnosing hardware

Memory testing for errors with memtest86+ as per KM1271513
Disk errors on console, in dmesg, /var/log/messages, chkdsk, iDRAC/iLO

RAID errors using commands in previous slide
CPU errors on console, in dmesg, /var/log/messages, ipmitool
M/B errors on console, in dmesg, /var/log/messages, ipmitool
PSU errors ipmitool

LCD front panel often displays error messages from SEL
46
General system performance issues

Slow UI response
log in via ssh (putty under Windows) as per KM1271560
check duplex mode and speed of the network interfaces using ethtool
verify that the Appliances own name is in the /etc/hosts file.
diagnose the processes consuming most CPU, memory and disk I/O with commands such as top, ps and
vmstat.
check disk space usage using commands such as df and du
check for errors in /var/log/messages and using the dmesg command
47
Types of RMA
Four key scenarios of RMA:
Single Failed Hard Drive replace drive
Multiple Failed Hard Drives replace entire appliance
Failed PSU (one or more) replace PSU
Internal hardware failure (CPU, RAM, M/B, RAID controller, etc) replace entire appliance
48
Filing an RMA
To avoid delays, be sure to provide all information below:
Contact Full Name
Company Name
Address (no PO boxes)
City, State, Postal Code
Country (if outside of US, must include)
Contact Phone Number:
VAT/Tax ID# (if outside of US, must include)
Serial Number of Defective Appliance:
Model:
ArcSight Version:
Technical Reason:
49
Work with ArcSight Support to review all data and file RMA with the above information.
Find out more

Attend these sessions
1045, The continuing
evolution of the HP
Arcsight CORR-Engine,
Tuesday 10:00 a.m.-10:50
p.m.
1034, From HP Arcsight
ESM to express migration ,
Tuesday, 4:00 p.m. 5:30
p.m.
Visit these demos

Solution building by
example, TT206
After the event

Contact your sales rep
SIEM now what?, TT211
Visit the Protect 724 website

https://protect724.arcsight.com
One size doesnt fit all:

customized training,
TT205
Download the Concepts whitepaper at:

https://protect724.arcsight.com/docs/D
OC-2174
Your feedback is important to us. Please take a few minutes to complete the session survey.
50
Thank you

1040 Bennacer Complete

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

1040 Bennacer Complete

Загружено:

Авторское право:

Доступные форматы

Managing an Arcsight

Express 3.0 appliance

Managing event archives

Managing ArcSight Express 3.0:

Connecting to Express 3.0

HP Integrated Lights-Out (iLO)

ArcSight Express 3.0 Management Console

ArcSight Express 3.0 Management Console

ArcSight Express 3.0 Management Console

AE 3.0 Advanced Administration

ArcSight Express 3.0 Process Management

Unified control of all ArcSight services with /sbin/service arcsight_services

ArcSight Express 3.0 Directory Layout

ArcSight Express 3.0 Storage Allocation

Total storage: 1.6TB

Event archival storage: 200GB

ArcSight Express 3.0 Storage Engines

InnoDB Built-in transactional support, allowing updates and deletes

ArcSight Express 3.0 Events

Remove side tables completely

ArcSight Express 3.0 DB Management

No event time correction(clamping) required

Managing ArcSight Express 3.0

Troubleshooting feature behavioral issues

Command line application check

Manager is not running

web service is available

web service is mixed_statuses

Logs & more

Reading events (channels/reports)

Behavior (startup / content)

Collect and upload logs to Service Center

Collecting application logs

Collecting database logs

Summary of Express 3.0 changes

Single script to Manage all ArcSight services

Elimination of descriptor Side Tables boosts performance

Managing ArcSight Express 3.0

Email notifications can be configured to alert people

Configuring event retention

The events portion of the CORR-Engine storage

Where the archives reside

Configuring archive schedule

Manual archive (user-driven)

...Still in active storage

2. No longer in active storage

Auditing and logs

ID Audit event description

Event archive settings updated

Event archive disk space used

Archive activation cancelled

Archive activation failed

Archive operation succeeded

Archive operation cancelled

Archive operation failed

Archive deactivation cancelled

Archive deactivation failed

Archive schedule cancelled

Archive schedule failed

We can see the Archive auditing in the Server.std.log and server.log

ArchiveSchedule4Display="Wed Dec 31 01:00:00 PST 1969"

<br>archiveDate=[Fri Sep 30 00:00:00 PDT 2011 --> 1317366000000millis]

How to restore archive backups to a new system

Managing ArcSight Express 3.0