Вы находитесь на странице: 1из 26

US 20040254919A1

(19) United States


(12) Patent Application Publication (10) Pub. No.: US 2004/0254919 A1
Giuseppini

(43) Pub. Date:

(54)

LOG PARSER

(75)

Inventor:

Dec. 16, 2004

Publication Classi?cation

Gabriele Giuseppini, Redmond, WA

(51)

Int. Cl.7 ..................................................... .. G06F 7/00

(US)

(52)

US. Cl. ................................................................ .. 707/3

Correspondence Address:
LEE & HAYES PLLC

(57)

ABSTRACT

421 W RIVERSIDE AVENUE SUITE 500


SPOKANE, WA 99201

(73) Assignee; MICROSOFT CORPORATIONRE};


MOND, WA (Us)

10/461,672

(22)

Jun. 13, 2003

described. In one aspect, a query against logged data is

has been designed to parse activity logs of multiple different


data formats. Responsive to receiving the query, the logged
data is parsed to generate query results. Output data is
created from the query results.

System Memory
1%
Application Programs LE
Log Parser

22

Query Engine
219
Other Application(s)
(E.g., OS, Runtime Services, a Client Application
such as Log Parser Library Host, Etc.)

EA
Program Data

LP Grammar-Based Query(ies)

&
Source Log File(s)
2%
Query Results

received. The query is based on a log parser grammar that

(21) Appl, No,:


Filed:

Systems and methods for parsing an activity log are

22
Output Data

(E.g., Fi|e(s), Database Table(s), ADO Objects, Etc.)


2%
Custom Plu -ln Reader

(E.g., a COM object)

Custom
Reader API

L1_8

m
Log Parser Library
Log Parser Common Library lLPCL)

(E.g., COM objects, Etc.)


2_2Q

API

Patent Application Publication Dec. 16, 2004 Sheet 1 0f 3

vmr

_
o8

_/

w?
_
_
~

xmOEm
ikDmE
_m wD

(um
j
i

work

.lw?rxi

"mm:

_.. |

mg.

a.

_/ $51852

US 2004/0254919 A1

Patent Application Publication Dec. 16, 2004 Sheet 2 0f 3

US 2004/0254919 A1

System Memory
1%
Agglication Programs 1_3_Q
Log Parser

202

Query Engine
212
Other Appiication(s)
(E.g., OS, Runtime Services, a Client Application
such as Log Parser Library Host, Etc.)

M
Program Data

LP Grammar-Based Query(ies)
2E

Source Log File(s)

2_08_
Query Results

22
Output Data

(E.g., File(s), Database Table(s), ADO Objects, Etc.)


2%
F

Custom

Custom (Plug-in) Reader

(E.g., a COM object)


m
F

Reader API

' m
Log Parser Library

Loq Parser Common Librarv (LPCQ

(E.g., COM objects, Etc.)


122

AP

. Q

Patent Application Publication Dec. 16, 2004 Sheet 3 0f 3


00

302 -\

Receive a Log Parser

Grammar-Based Query
Against Logged Data

Parse the Logged Data


304

Based on Log Parser

W Grammar Specified by the


Query to Generate a Query
Result

Generating Output Data from


306 \ the Query Results, the Output
Data Being Associated with
One or More Output Targets

US 2004/0254919 A1

Dec. 16, 2004

US 2004/0254919 A1

LOG PARSER

[0008]

FIG. 2 is a block diagram that shoWs further

exemplary aspects of system memory of FIG. 1, including


TECHNICAL FIELD

[0001] The invention pertains to data processing.

application programs and program data for log parser.

[0009] FIG. 3 shoWs an exemplary procedure for log


parser. In one implementation, the operations of FIG. 3 are

BACKGROUND

implemented by the log parser 202 of FIG. 2. In another


implementation, the operations of FIG. 3 are implemented

[0002] Activity logs are commonly used by system admin

by a third-party application that interfaces With one or more

istrators to record events of interest. The type of information

Common Object Model (COM) objects exposed by a log

stored in any activity log is generally a function of the


purpose of the monitoring application/tool used to generate
and maintain the log. That is, different monitoring tools are

parser common library of FIG. 2.


DETAILED DESCRIPTION

generally used to generate activity logs for different types of


system activity. For instance, one monitoring tool may log

[0010] OvervieW

Web site traf?c, another tool used to monitor Intranet activ


ity, yet another tool used to record information associated

beloW. The log parser is a versatile tool that runs Structured

With exception handing, computer system performance,

[0011]

Systems and methods for a log parser are described

and/or the like. Thus, the particular monitoring tool(s) that

Query Language (SQL)-type queries against source ?les


(i.e., log ?les) to implement many activity log related tasks).
SQL-type queries (i.e., the LogParsers SQL queries) do not

is/are to be used to log data is based on the type(s) of


monitoring to be performed (i.e., the events to be moni

folloW exactly the ANSI standard for the SQL language. In


addition, Log Parsers SQL queries add some elements for

tored).

tasks not available in the ANSI standard. Such tasks include,

resource accesses, ?le generation and modi?cation events,

[0003] Respective ones of multiple different activity log


ging tools generally output data (logged data) in any of
multiple possible document and data formats. Such data

formats include, for example, third-party proprietary data


format(s), comma-separated value (CSV), Extensible
Markup Language (XML), ASCII text, World Wide Web
Consortium (W3C), Internet Information Service (IIS), and/
or other data formats. Since a administrator Will typically

need multiple activity logging tools to adequately monitor

for example, importing, parsing, presenting, and exporting


many different input log ?le data formats (e.g., CSV, XML,
text, W3C, IIS, database table, WINDOWS event logging,
and other data formats). Additionally, the log parser provides
for ?ltering log entries, searching for data and patterns in
?les of various data formats, converting log ?les from one
data format to another data format, creation of formatted
reports and XML ?les containing data retrieved from dif
ferent log sources, exporting data (all or selected portions of

application, system, netWork, and or other events, the system


administrator Will also require multiple custom-built tools to

log ?les) to database tables (e.g., SQL tables), data mining,

parse, present/vieW, and/or export the resulting logged data,

[0012] To these ends, the log parser extracts records, using

Which is typically of different data formats. This is a

one or more SQL-type queries, from source ?les of various

and so on.

substantially onerous requirement, especially in vieW of the

input source types. The log parser query engine processes

many different types of events that generally need to be


logged, and in vieW of the diverse data formats typically

these records?ltering, grouping, and ordering them

output by respective ones of the logging tools.

according to the conditions speci?ed in the SQL-type query.


Log parser then presents the processed records (i.e., the

[0004] Accordingly, systems and methods that do not

query results) to an end-user, and/or Writes the query results

require use of multiple speci?cally designed and indepen


dent tools to parse, present/vieW, and/or export activity logs
of multiple different respective data formats are greatly

or more selected data formats supported by the log parser.

desired.

to one or more target output ?les or database tables in one

[0013] In this manner the log parser makes it possible to


request information from log ?les of almost any data format

and produce the desired information (i.e., the query results)


SUMMARY

[0005] Systems and methods for parsing an activity log are


described. In one aspect, a query against logged data is
received. The query is based on a log parser grammar that

has been designed to parse activity logs of multiple different


data formats. Responsive to receiving the query, the logged
data is parsed to generate query results. Output data is
created from the query results.
BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the ?gures, the left-most digit of a component


reference number identi?es the particular ?gure in Which the
component ?rst appears.
[0007]

FIG. 1 is a block diagram of an exemplary com

puting environment Within Which systems and methods for


log parser may be implemented.

for presentation and/or storage in a ?le of almost any data


format or into an SQL database. Thus, log parser addresses

the limitations of conventional activity log interfacing tech

niques that require multiple speci?cally designed and inde


pendent tools to parse, present/vieW, and/or export activity
logs of multiple different respective data formats. These and
other aspects of the log parser, including the exemplary
operating environment of FIG. 1 and exemplary log parser
grammar for generating the SQL-type queries are noW
described in greater detail.

[0014] Exemplary Operating Environment


[0015] Turning to the draWings, Wherein like reference
numerals refer to like elements, the invention is illustrated as

being implemented in a suitable computing environment.


Although not required, the invention is described in the
general context of computer-executable instructions, such as
program modules, being executed by a personal computer.

Dec. 16, 2004

US 2004/0254919 A1

Program modules generally include routines, programs,


objects, components, data structures, etc., that perform par
ticular tasks or implement particular abstract data types.
[0016]

FIG. 1 illustrates an example of a suitable com

modules that are immediately accessible to and/or presently

being operated on by processor 104.


[0020]

Computer 102 may further include other remov

able/non-removable, volatile/non-volatile computer storage

puting environment 100 on Which the subsequently


described systems, apparatuses and methods for log parser

media. For example, FIG. 1 illustrates a hard disk drive 116


for reading from and Writing to a non-removable, non

may be implemented (either fully or partially). Exemplary

volatile magnetic media (not shoWn and typically called a


hard drive), a magnetic disk drive 118 for reading from
and Writing to a removable, non-volatile magnetic disk 120
(e.g., a ?oppy disk), and an optical disk drive 122 for
reading from or Writing to a removable, non-volatile optical

computing environment 100 is only one example of a


suitable computing environment and is not intended to
suggest any limitation as to the scope of use or functionality

of systems and methods the described herein. Neither should


computing environment 100 be interpreted as having any

disk 124 such as a CD-ROM/R/RW, DVD-ROM/R/RW/+

tion of components illustrated in computing environment

R/RAM or other optical media. Hard disk drive 116, mag


netic disk drive 118 and optical disk drive 122 are each

100.

connected to bus 108 by one or more interfaces 126.

[0017] The methods and systems described herein are


operational With numerous other general purpose or special
purpose computing system environments or con?gurations.

[0021] The drives and associated computer-readable


media provide nonvolatile storage of computer readable
instructions, data structures, program modules, and other
data for computer 102. Although the exemplary environment

dependency or requirement relating to any one or combina

Examples of Well knoWn computing systems, environments,


and/or con?gurations that may be suitable for use include,
but are not limited to, personal computers, server computers,

multiprocessor systems, microprocessor-based systems, net


Work PCs, minicomputers, mainframe computers, distrib
uted computing environments that include any of the above
systems or devices, and so on. Compact or subset versions

of the frameWork may also be implemented in clients of


limited resources, such as handheld computers, or other

computing devices. The invention may also be practiced in


distributed computing environments Where tasks are per

formed by remote processing devices that are linked through


a communications netWork. In a distributed computing

environment, program modules may be located in both local


and remote memory storage devices.

[0018] As shoWn in FIG. 1, computing environment 100


includes a general-purpose computing device in the form of
a computer 102. The components of computer 102 can
include, by are not limited to, one or more processors or

processing units 104, a system memory 106, and a bus 108

that couples various system components including system


memory 106 to processor 104. The system bus 108 repre
sents one or more of any of several types of bus structures,

including a memory bus or memory controller, a peripheral


bus, an accelerated graphics port, and a processor or local

bus using any of a variety of bus architectures. By Way of


example, and not limitation, such \-architectures include

Industry Standard Architecture (ISA) bus, Micro Channel


Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video
Electronics Standards Association (VESA) local bus, and
Peripheral Component Interconnects (PCI) bus also knoWn
as MeZZanine bus.

[0019]

Computer 102 typically includes a variety of com

puter readable media. Such media may be any available

media that is accessible by computer 102, and it includes


both volatile and non-volatile media, removable and non
removable media. In FIG. 1, system memory 106 includes
computer readable media in the form of volatile memory,
such as random access memory (RAM) 110, and/or non
volatile memory, such as read only memory (ROM) 112. A

basic input/output system (BIOS) 114, containing the basic


routines that help to transfer information betWeen elements
Within computer 102, such as during start-up, is stored in
ROM 112. RAM 110 typically contains data and/or program

described herein employs a hard disk, a removable magnetic


disk 120 and a removable optical disk 124, it should be
appreciated by those skilled in the art that other types of
computer readable media Which can store data that is

accessible by a computer, such as magnetic cassettes, ?ash


memory cards, digital video disks, random access memories

(RAMs), read only memories (ROM), and the like, may also
be used in the exemplary operating environment.
[0022] A user may provide commands and information
into computer 102 through input devices such as keyboard
140 and pointing device 142 (such as a mouse). Other

input devices (not shoWn) may include a microphone, joy


stick, game pad, satellite dish, serial port, scanner, camera,
etc. These and other input devices are connected to the

processing unit 104 through a user input interface 144 that


is coupled to bus 108, but may be connected by other
interface and bus structures, such as a parallel port, game
port, or a universal serial bus (USB).

[0023]

A monitor 146 or other type of display device is

also connected to bus 108 via an interface, such as a video

adapter 148. In addition to monitor 146, personal computers

typically include other peripheral output devices (not


shoWn), such as speakers and printers, Which may be con

nected through output peripheral interface 150.


[0024]

Computer 102 may operate in a netWorked envi

ronment using logical connections to one or more remote


computers, such as a remote computer 152. Remote com

puter 152 may include many or all of the elements and

features described herein relative to computer 102. Logical


connections shoWn in FIG. 1 are a local area netWork (LAN)
154 and a general Wide area netWork
156. Such

netWorking environments are commonplace in of?ces, enter

prise-Wide computer netWorks, intranets, and the Internet.


[0025] When used in a LAN netWorking environment,
computer 102 is connected to LAN 154 via netWork inter
face or adapter 158. When used in a WAN netWorking
environment, the computer typically includes a modem 160
or other means for establishing communications over WAN

156. Modem 160, Which may be internal or external, may be


connected to system bus 108 via the user input interface 144
or other appropriate mechanism. Depicted in FIG. 1, is a
speci?c implementation of a WAN via the Internet. Here,

Dec. 16, 2004

US 2004/0254919 A1

computer 102 employs modem 160 to establish communi


cations With at least one remote computer 152 via the
Internet 162.

[0026]

In a networked environment, program modules

depicted relative to computer 102, or portions thereof, may


be stored in a remote memory storage device. Thus, e.g., as

depicted in FIG. 1, remote application programs 164 may


reside on a memory device of remote computer 152. It Will

be appreciated that the netWork connections shoWn and


described are exemplary and other means of establishing a

communications link betWeen the computers may be used.


[0027]

A number of program modules may be stored on

the hard disk, magnetic disk 120, optical disk 124, ROM
112, or RAM 110, including, e.g., an operating system (OS)
128 to provide a run-time environment, application pro
grams 130 for log parser, other program modules 132 (e.g.,
device drivers, etc.), and program data 134 such source
code, log ?le queries based on log parser grammar, inter
mediate data, and/or the like.
[0028] FIG. 2 is a block diagram that shoWs further
exemplary aspects of system memory 106 of FIG. 1, includ

ing application programs 130 and program data 134 for log
parser. In this implementation, application programs 130
include, for example log parser 202 and other applications
204 such as the operating system (OS) 128 of FIG. 1, and

(>), IS NULL, LIKE, and IS IN. And the Log Parser SOL
engine supports most standard SQL query clauses: SELECT,
WHERE, GROUP BY, HAVING, and ORDER BY.
[0033] In one implementation, the log parser 202 alloWs
third-party softWare developers to add-value to the log
parser 202 via one or more plug-ins. For example, to read
and parse an input source log ?le 208 of a particular data

format, the log parser Will interface With a plug-in such as


the custom reader 216. In this implementation, the custom

reader 216 is a Common Object Model (COM) object that


exposes its operational capabilities via an Application Pro
gram Interface (API) 218 that is designed to interface With
the log parser 202.

[0034] To generate query result(s) 212, the query engine


210 may perform any combination of log entry (i.e., record)
extraction and/or ?ltering operations, searching for data
and/or patterns in ?les of various data formats, grouping
and/or ordering extracted information according to the con
ditions speci?ed in the query 206. The log parser 202
generates output data 214 from the query results 212. The
output data 214 may represent the end results of converting
log ?les from one data format to another data format,
creation of formatted reports and XML ?les containing data
retrieved from different log sources, exporting data (all or

selected portions of log ?les) to database tables (e.g., SOL

a runtime to provide the log parser 202 With services such as

tables), data mining, and so on. For example, converting

Just-In-Time (JIT) compiling, memory management, and so

log ?les is the effect of running a query 206 on format A,

on. The log parser 202 alloWs a user or executed script to

assert/run/execute a log parser grammar-based query 206,


Which is hereinafter often simply referred to as a query,
against one or more source log ?les 208. Exemplary log
parser grammar is described in greater detail beloW in
reference to TABLES 1-17.

generating query result(s) 212 and Writing the result(s) to


format B. The creation of formatted reports and export
ing data to database tables is accomplished in analogous

operations.
[0035] The log parser 202 presents query result(s) 212 to
an end-user (e.g., via the display monitor 146 of FIG. 1),
Writes the query result(s) one or more database tables, and/or

[0029] Responsive of receiving a query 206 against a log


?le 208, the query engine 210 portion of the log parser 202
parses the log parser grammar-based query 206 to generate

Writes the query result(s) into data ?le(s) of speci?ed data

query result(s) 212, Which represent the desired/queried-for

results(s) 212 that have been exported to database tables,

information. Query 206 may specify any of numerous dif


ferent functions for the query engine 212 to perform With

?les, and so on, are represented as output data214.

respect to the speci?ed source log ?le(s) 208. Exemplary

format, and so on. For purposes of discussion, query

[0036] Exemplary Log Parser Grammar

such functions are described in greater detail beloW in


reference to TABLE 2. As indicated above, source log
?les(s) 208, or log(s) can be in any of numerous different

[0037] Table 1 lists exemplary grammar used by the query

document and data formats, for instance, CSV, XML, text,


W3C, IIS, database table, WINDOWS event logging, and/or

TABLE 1

engine 210.

EXEMPLARY LOG PARSER QUERY ENGINE GRAMMAR

other data formats.

[0030] For example, the query engine 210 supports a


dialect of Structured Query Language (SQL) syntax, as
described above With respect to the term SOL-type. The

Element
<query>

query engine 210 treats an input source 208 as a relational

table, so ?elds act as table columns and each ?eld is assigned


a data type. Data types can be STRING, INTEGER, REAL,
and TIMESTAMP. Any value can assume the NULL value.
Aquery 206 can embed ?elds in functions, such as STRCAT,
STRLEN, and REVERSEDNS, and can nest functions mul

tiple times. For example:

[0031] SUBSTR(cs-uri-stem,

INDEX_OF(cs-uri

stem, TO_STRING(sc-status))).
[0032] In addition, the query engine 210 supports the
aggregate functions SUM, COUNT, AVG, MIN, and MAX.
It supports the most common operators, such as greater than

<selecticlause>

Syntax
<selecticlause> <fromiclause>

[<toiclause>]
[<Whereiclause>] [<groupibyiclause>]
[<havingiclause>] [<orderibyiclause>]
SELECT [TOP<integer>] [DISTINCTI
ALL] <selectionilist> |
SELECT [TOP<integer>] [DISTINCTI
ALL] *

<selectionilist>

<selectionilistiel> |

<selectionilistiel>

<selectionilistiel>, <selectionilist>
<?eldiexpr> AS <alias> |

<fromiclause>
<toiclause>

TO <toientity> |

<Whereiclause>

WHERE <expression> |

FROM <fromientity>

Dec. 16, 2004

US 2004/0254919 A1

TABLE l-continued

TABLE l-continued

EXEMPLARY LOG PARSER QUERY ENGINE GRAMMAR

EXEMPLARY LOG PARSER QUERY ENGINE GRAMMAR

Element

Syntax

Element

Syntax

<expression>

<term1> OR <expression> I
<term1>

<timestamp>

TMESTAMP (<stringivalue>,
<timestampiformat>)

<term1>

<term2> AND <term1> I


<term2>

<timestampiformat>

<timestampiseparator>
O*7(<timestampielement>

<term2>

<?eldiexpr>
<?eldiexpr>
<?eldiexpr>
<?eldiexpr>

<reliop> <value> I
LIKE <li_keivalue> I
<unaryiop> I
<incliop> <content> I

<timestampielement>

<?eldiexpr> <reliop> ALLIANY

l*2(HIh)I

<content> I
(<?eldiexprilist>) <incliop> <content> I

1*2 m I
1*2 s

(<?eldiexprilist>) <reliop>

<timestampiseparator>

ALLIANY <content> I

<content>

<anyichariexceptitimestampielement>
<null>

NOT <t6TIT12> I

<li_keivalue>

*(<anyfchar>|%|f)

(<eXPreSSiOn>)

<stringivalue>

*(<anyichar>)

(<valueilist>) I

(<query>)

<groupibyiclause>
h _

<timestampiseparator>)
1*4 y I
1*4 M I
1*4 d I

< avmufc ause>


<orderibyiclause>

<?eldiexprilist>

GROUP BY <?eldiexprilist> I
<nu11>

:QIILNG <expresslon> I
ORDER BY <?eldiexprilist> [ASCI
DESCH

[0038]

Table 2 lists exemplary functions that can be

directed to the log parser 202.

TABLE 2

ORDER BY * [ASCIDESC] I
<I11l11>

Log Parser Functions

<?eldiexpr> |

SUBSTR(string <STRING>, start <INTEGER> [, length <INTEGER>])

STRCAT(string1 <STRING>, stringZ <STRING>)


STRLEN(string <STRING>)
STRREV(string <STRING>)
TOiINT(argument <any type>)
This function converts the speci?ed argument to an integer. If the
argument cannot be converted, the function returns NULL.

<sqlfunctioniexpr>

TOiREAL(argument <any type>)

TOiSTRING(argument <INTEGERIREAL>)I
(timestamp <TIMESTAMP>, format <STRING>)
<functio niexpr>

TOiDATE(timestamp <TIMESTAMP>)

<?eldilist>

This function transforms the speci?ed argument into a timestamp

<valueilist>

TOiTIME(timestamp <TIMESTAMP >)

containing date values only.


This function transforms the speci?ed argument into a timestamp

<valueilistiroW>

<value> I
<value>, <valueilistiroW>

<sqlfunction>

SUMIAVGIMAXIMIN

<function>

STRCATISUBSTRISTRREVITOLINT |

TOLREALITOLSTRINGITOLDATE I
TOiTIME |

TOLTIMESTAMPITOLHEXI

<TIMESTAMP>)|
(string <STRING>, format <STRING>)
The ?rst example combines tWo timestamps containing date and time
values into a single timestamp. The second example parses a string into a

timestamp, according to the timestamp pictures de?ned in the second

REPLACELSTR I REPLACELCHR I

argument.

STRLENIINDEXLOFILASTLINDEXLOF I
ADDISUB |
DIVIMULIREPLACELIFLNULL |

This function returns the hexadecimal string representation of the integer

UNIQUEiIDISYSTEMLTIMESTAMP I

replaceString <STRING>)

SYSTEMLTIMEISYSTEMLUTCOFFSET I

replaceString <STRING>)

REPLACELIFLNOTLNULL I

SYSTEMiDATE I

TOLLOCALTIME I

TOLUTCTIMEITOLLOWERCASE I
TOiUPPERCASE I

QUANTIZEIREVERSEDNSIURLESCAPE I

URLUNESCAPE I

TOiHEX(argument <INTEGER>)
argument.

REPLACELSTR( string <STRING>, searchString <STRING>,


REPLACE CHR( string <STRING>, searchCharacters <STRING>,

INDEXiOF(string <STRING>, searchStr <STRING>)


LASTiINDEXiOF(string <STRING>, searchStr <STRING>)
ADD(argument1 <any type>, argument2 <any type>)
SUB(argument1 <any type>, argument2 <any type>)

SQRISQRROOTILOGIEXP |

DIV(argument1 <INTEGERIREAL>, argument2 <INTEGERIREAL>)


MUL(argument1 <INTEGERIREAL>, argument2 <INTEGERIREAL>)
REVERSEDNS(ipAddress <STRING>)

EXTRACTLTOKENIRESOLVELSID

If the argument does not specify a valid IP address (IPv4 or IPv6), or if


the
IP address cannot be resolved, the result is the argument string itself.

<integer> I
<timestamp> I

This function replaces the speci?ed argument Whenever it has a NULL


value.

EXTRACTLVALUE I
WINSZLERRORLDESCRIPTION I

<value>

containing time values only.


TOiTIMESTAMP(dateTime1 <TIMESTAMP>, dateTimeZ

<stringivalue> I
<real> I

NULL

REPLACEiIFiNULL(argument <any type>, replaceValue <any type>)


REPLACEiIFiNOTiNULL(argument <any type>, replaceValue <any

typ?)
<incliop>

ININOT IN

This function replaces the speci?ed argument Whenever it has a value

<unaryiop>

IS NULLIIS NOT NULL

other than NULL.

Dec. 16, 2004

US 2004/0254919 A1

TABLE 2-continued

SYSTEMLTIMESTAMP( )
SYSTEMLDATE( )
SYSTEMLTIME( )
SYSTEMiUTCOFFSET( )

TABLE 3-continued
EXEMPLARY TIMESTAMP ELEMENTS

Timestamp
Element

Description

h,H
hh,HH

hour Without leading Zeros


hour With leading Zeros

m
mm
s
ss

minutes
minutes
seconds
seconds

URLESCAPE(url <STRING> [, codepage <INTEGER>])

1
ll

milliseconds Without leading Zeros


milliseconds With leading Zeros

This function returns the HEX encoding (as speci?ed in RFC2396) of the
argument passed. The codepage used by default is UTF-8.

n
nn

nanoseconds Without leading Zeros


nanoseconds With leading Zeros

This function returns the absolute value of the current time Zone offset.

TOiLOCALTIME(timestamp <TIMESTAMP>)
TOiUTCIIME(timestamp <TIMESTAMP>)
TOiLOWERCASE(string <STRING>)
TOiUPPERCASE(string <STRING>)
UNIQUEID ([startValue <INTEGER>])
This function returns a unique INTEGER value every time a roW is

generated. The default start value is 1.

Without leading Zeros


With leading Zeros
Without leading Zeros
With leading Zeros

URLUNESCAPE(url <STRING> [, codepage <INTEGER>])


This function returns the HEX un-encoding (as speci?ed in RFC2396) of
the argument passed. The codepage used by default is UTF-8.

SQR(argument <INTEGER | REAL>)


SQRROOT(argument <INTEGER | REAL>)
LOG(argument <INTEGER | REAL>)
EXP(argument <INTEGER | REAL>)
QUANTIZE(argument <INTEGER | REAL | TIMESTAMP>,
QUANTIZATION<INTEGER | REAL>)

[0040]

Table 4 lists the Wildcard characters used by the

<like_value> operand of the LIKE operator.


TABLE 4
EXEMPLARY WILDCARD CHARACTERS

This function rounds the speci?ed value to the loWest sub-multiple of the

quantization value. When used With timestamps, the quantization argument


refers to the number of seconds.

EXTRACTiVALUE(argument <STRING>, key <STRING> [, separator

<STRING>])
This function parses a list of valuename = value strings separated by the

character passed as the separator argument and returns the value portion
identi?ed by the key argument. The separator value has a default value of
8a. For example:

Character

Use For

Any string

Any character

\%
\\

The % character
The \ character

\<anyicharacter>

The speci?ed character

EXTRACILVALUE( siteID=example.com&countrycode=usa,
countrycode)returns usa.

WIN3ZiERRORiDESCRIPTION<WII132EITOTCOCI6 <INTEGER> )
This function returns a string containing the WINDOWS error message

represented by the speci?ed error code.

EXTRACTiTOKEN(argument <STRING>, index <INTEGER>[,


separator <STRING>]) This function parses a list of strings separated by
the separator argument string and returns the portion identi?ed by the 0
based index argument. The separator value has a default value of ,. For

example:EXTRACIfTOKENCvaluel,value2,value3,value4, 2)
returns value3.

[0041] To specify Unicode characters in <string_value>


literals, type them in the following notation: \unnnn,
Where nnnn refers to the four-digit hexadecimal representa
tion of the Unicode character. For example, to specify a TAB

character, type the following: \u0009.\


[0042] Table 5 lists the escape characters accepted by log
parser 202 When parsing <string_value> literals.

RESOLVEiSID( sid <STRING> [, computerName <STRING>])


This function returns the fully speci?ed account name represented by the

TABLE 4

argument SID. If the argument doesnt specify a valid SID, or if the SID
cannot be resolved, the function returns the SID string itself. The optional
computerName argument speci?es the computer on Which to perform the
account lookup.

[0039] Table 3 lists exemplary log parser 202 timestamp

EXEMPLARY ESCAPE CHARACTERS

Escape sequence

Converted to

\\

The character

The

character

elements.
TABLE 3
EXEMPLARY TIMESTAMP ELEMENTS

Timestamp

[0043] To specify hexadecimal values, use the 0x pre?x.


For example: 0x000f2.

[0044] Exemplary Source Log File Input Data Formats


[0045] This implementation of the log parser 202 supports

Element

Description

y
yy
yyy
yyyy

1-digit
2-digit
3-digit
4-digit

M
MM

month as digit Without leading Zeros


month as digit With leading Zeros

[0047]

MMM
MMMM

month as 3-character abbreviation of month name


month as full month name

d
dd
ddd
dddd

day
day
day
day

[0048] IISMSID: This is the log format for ?les


generated by IIS When the MSIDFILT ?lter or the

as
as
as
as

year
year
year
year

digit Without leading Zeros


digit With leading Zeros
3-character abbreviation of day name
full day name

the folloWing source log ?le 208 input data formats:


[0046]

IISW3C: This is the IIS W3C Extended log

?le format.

IIS: This is the IIS log ?le format.

CLOGFILT ?lter is installed.

[0049] NCSA: This is the IIS NCSA Common log ?le


format.

Dec. 16, 2004

US 2004/0254919 A1

[0050] ODBC: This is the IIS ODBC format, Which


sends log ?les to an ODBC-compliant database.

TABLE 6-continued

[0051] BIN: This is the IIS binary log ?le format.

Field

Data Type

[0052]

s-total-procs
s-active-procs
s-stopped-procs

INTEGER
INTEGER
INTEGER

URLSCAN: This is the format for URLScan

logs.
[0053] HTTPERR: This is the IIS 6.0 HTTP error log
?le format.

[0054]

EVT: This is the Microsoft WINDOWS Event

Messages format.

[0063]

Fields that are not logged in the log ?le are returned

as NULL.

[0064] The IISW3C input data format accepts the folloW


ing values in the FROM statement:

[0055] TEXTWORD: This is a generic teXt ?le,


Where the TEXT value is any separate Word.
[0056] TEXTLINE: This is a generic teXt ?le, Where
the TEXT value is any separate line.
[0057]

CSV: This is a comma-separated list of values.

[0065] A ?le name, or a comma-separated list of ?le


names, including names that contain Wildcards, such
as LogFiles\W3SVC3\eX*.log.

[0066] An Active Directory Services Interface (ADSI)


path, or a comma-separated list of paths, specifying the
virtual site or site name that hosts one or more logs ?les and

[0058] W3C: This is a generic W3C log ?le, such as


a log generated by WINDOWS Media Services or
Personal FireWall.

[0059]

FS: This provides information about ?le and

directory properties.

The log parser 202 can query any IIS log ?le data

formats.

[0061]

PUTER/W3SVC/2>, <//FARM\W3SVC/WWW.s5.com> or

<4, 9> When referring to the local computer.

[0067] The stdin command, used to pipe command


eXecutions, such as type eXtend1.log|LogParser

IIS Log File Formats

[0060]

Whose log ?le(s) are to be parsed, enclosed in angle brackets


(< >), such as <//MYCOMPUTER/W3SVC/1, //MYCOM

SELECT * from stdini:IISW3C.

[0068] The following options are available for the IISW3C


input data format:

[0069] iCodepage: Speci?es the codepage in Which

1. nswsc

the ?les are encoded; legal values are 1252, 0 (cur

[0062] This input data format parses IIS W3C EXtended


log ?les 208. Table 6 lists IISW3C ?elds and corresponding
data types.
TABLE 6
Field

Data Type

LogFilename
LogRoW

STRING
INTEGER

date
time

TIMESTAMP
TIMESTAMP

c-ip

STRING

cs-username

STRING

s-sitename

STRING

s-computername

STRING

s-ip

STRING

s-port

INTEGER

cs-method
cs-uri-stem

STRING
STRING

rent system codepage), 1 (Unicode), 65001 (UTF


8), and so on. The default value is 2, meaning that
log parser 202 determines the codepage based on the
?le name and the IIS metabase settings.

[0070] dQuotes: Speci?es that the strings in the ?le


should be enclosed in quotation marks (). Legal
values are ON or OFF. The default value is OFF.

[0071] dirTime: Instructs the tool to return the #Date:


directive as date/time When the date/time ?elds in the
log ?le are NULL. Legal values are ON or OFF. The
default value is OFF.

[0072] 2. 11s
[0073] This input data format parses the Microsoft IIS log
format ?les generated by IIS. Table 6 lists the IIS input data
format ?elds and corresponding data types.

cs-uri-query

STRING

sc-status

INTEGER

sc-substatus
sc-Win32-status

INTEGER
INTEGER

sc-bytes
cs-bytes

INTEGER
INTEGER

Field

Data Type

time-taken
cs-version
cs-host

INTEGER
STRING
STRING

LogFileName
LogRoW

STRING
INT'EGER

cs(User-Agent)

STRING

STRING
STRING
TIMESTAMP
TIMESTAMP
STRING
STRING
STRING
INT'EGER

INT'EGER

TABLE 7

s-process-type

STRING

s-user-time
s-kernel-time

REAL
REAL

UserIP
UserName
Date
Time
ServiceInstance
HostName
ServerIP
TimeTaken

s-page-faults

INTEGER

BytesSent

cs(Co okie)

STRING

cs(Referer)

STRING

s-event

STRING

Dec. 16, 2004

US 2004/0254919 A1

TABLE 7-continued

TABLE 8-continued

Field

Data Type

Field

Data Type

BytesReceived

INTEGER

Target

STRING

Status Co de
Win32StatusCode
RequestType

INTEGER
INTEGER
STRING

UserAgent
Referrer
GUID

STRING
STRING
STRING

Target

STRING

PassportID

STRING

Parameters

STRING

PartnerID
Parameters

STRING
STRING

[0074] The IIS input data format accepts the following


values in the FROM clause:
[0075]

A ?le name, or a comma-separated list of ?le

names, including names containing Wildcards, such


as LogFiles\W3SVC3\in02* .log.
[0076] An ADSI path, or a comma-separated list of
paths, specifying the virtual site or site name Whose
log ?les are to be parsed, enclosed between angle
brackets (<>), such as <//MYCOMPUTER/W3SVC/

1,

//MYCOMPUTER/W3SVC/2>,

<//FARM/

W3SVC/WWW.s5.com> or <4, 9> When referring to


the local computer.

[0077] The stdin command, used to pipe command


executions, such as type inetsv1.log|LogParser
SELECT * from stdin-i:IIS.

[0078] The folloWing options are available for the IIS


input data format:

[0079] iCodepage: Speci?es the codepage in Which


the ?les are encoded; legal values are 1252, 0 (cur

rent system codepage), 1 (Unicode), 65001 (UTF


8), and so on. The default value is 2, meaning that
log parser 202 determines the codepage based on the
?lename and the IIS metabase settings.
[0080] locale: Speci?es the locale to use When pars
ing the ?les date and time values. Legal values are
locale IDs (such as 1033) or locale names (such as
JPN). The default value is the current system locale.

[0081] 3. IISMSID
[0082] This input data format parses the Microsoft IIS log
format ?les generated by IIS When the MSIDFILT ?lter or
the CLOGFILT ?lter is installed. Table 8 lists the IISMSID

input data format ?elds and corresponding data types.


TABLE 8

[0083] The IISMSID input data format accepts the fol


loWing values in the FROM clause:
[0084] A ?le name, or a comma-separated list of ?le
names, including names that contain Wildcards, such
as LogFiles\W3SVC3\inetsv*.log.

[0085] An ADSI path, or a comma-separated list of


paths, specifying the virtual site or site name Whose
log ?les are to be parsed, enclosed in angle brackets

(< >), such as<//GABRIEGIO/W3SVC/1, //GABR


IEGI1/W3SVC/7>,

<//FARM/W3SVC/

WWW.s5.com>, or <4,9> When referring to the local

computer.
[0086] The stdin command, used to pipe command execu
tions, such as type inetsv2.log|LogParser SELECT * from
stdin-i:IISMSID.

[0087] The folloWing options are available for the IISM


SID input data format:
[0088] clog?lt: Instructs log parser 202 to use the
CLOGFILT value separator convention When pars
ing the supplied log ?les; legal values are ON or
OFF. The default value is OFF.

[0089] iCodepage: Speci?es the codepage in Which


the ?les are encoded; legal values are 1252, 0 (cur

rent system codepage), 1 (Unicode), 65001 (UTF


8), and so on. The default value is 2, meaning that
log parser 202 determines the codepage based on the
?lename and the IIS metabase settings.
[0090] locale: Speci?es the locale to use When pars
ing the ?les date and time values; legal values are
locale IDs (such as 1033) or locale names (such as
JPN). The default value is the current system locale.

[0091] 4. NCSA

Field

Data Type

LogFileName
LogRoW

STRING
INTEGER

UserIP
UserName
Date
Time
ServiceInstance
HostName
ServerIP
TimeTaken

STRING
STRING
TIMESTAMP
TIMESTAMP
STRING
STRING
STRING
INTEGER

BytesSent
BytesReceived

[0092] This input data format parses NCSA Common log


?les. Table 9 lists the NCSA ?eld names and corresponding

data types.
TABLE 9
Field

Data Type

LogFileName
LogRoW

STRING
INTEGER

INTEGER
INTEGER

RemoteHostName

STRING

RemoteLogName

STRING

StatusCode
Win32StatusCode

INTEGER
INTEGER

UserName
DateTime

STRING
TIMESTAMP

RequestType

STRING

Request

STRING

Dec. 16, 2004

US 2004/0254919 A1

TABLE 9-continued
Field

Data Type

Status Co de

INTEGER

BytesSent

INTEGER

[0093] The NCSA input data format accepts the following


values in the FROM clause:

[0103] An ADSI path, or a comma-separated list of


paths, specifying the virtual site or site name Whose
log ?les are to be parsed, enclosed in angle brackets

(<>), such as<//GABRIEGIO/W3SVC/1, //GABR


IEGI1/W3SVC/7>,
<//FARM/W3SVC/
WWW.s5.com> or <4, 9>When referring to the local
machine.

[0104] There are no options available for the ODBC input


data format.

[0094] A ?le name, or a comma-separated list of ?le


names, including names that contain Wildcards, such
as LogFiles\W3SVC3\ncsa2*.log.

[0105]

[0095] An ADSI path, or a comma-separated list of


paths, specifying the virtual site or site name Whose
log ?les are to be parsed, enclosed in angle brackets
(<>) such as <//GABRIEGIO/W3SVC/1, //GABR

requests received by all the virtual sites on the same server


running IIS 6.0. Table 11 lists the BIN ?eld names and

IEGI1//W3SVC/7>,

<//FARM/W3SVC/

6.0 BIN

[0106] This input data format reads the central binary log
?les generated by IIS 6.0. These log ?les contain all the

corresponding data types.

WWW.s5.com> or <4, 9> When referring to the local


TABLE 11

computer.
[0096] The stdin command, used to pipe command
executions, such as type ncsa1.log|LogParser
SELECT * from stdin-i:NCSA.

[0097] The option available for the NCSA input data


format is as folloWs: iCodepage: Speci?es the codepage in
Which the ?les are encoded; legal values are 1252, 0 (current
system codepage), 1 (Unicode), 65001 (UTF-8), and so on.
The default value is 2, meaning that log parser 202
determines the codepage based on the ?lename and the IIS

metabase settings.
[00%] 5. ODBC
[0099] This input data format reads the ?elds directly from
the SQL table populated by IIS When the Web Server is
con?gured to log to an ODBC target. Table 10 lists the
ODBC input data format ?eld names and corresponding data

Field

Data Type

LogFileName

STRING

RecordNumber

INTEGER

ComputerName

STRING

SiteID
DateTime

INTEGER
TIMESTAMP

ClientIpAddress
ServerIpAddress

STRING
STRING

ServerPort
Method
Pro to colVers ion
ProtocolStatus
SubStatus
TimeTaken

INTEGER
STRING
STRING
INT'EGER
INT'EGER
INT'EGER

BytesSent
BytesReceived

INT'EGER
INT'EGER

Win32Status
UriStem

INT'EGER
STRING

UriQuery

STRING

UserName

STRING

types.
TABLE 10

[0107] The BIN input data format accepts the folloWing


values in the FROM clause:

Field

Data Type

ClientHost
UserName

STRING
STRING

LogTime

TIMESTAMP

Service
Machine
ServerIP

STRING
STRING
STRING

[0109] An ADSI path, or a comma-separated list of paths,

ProcessingTime
BytesRecvd
BytesSent

INTEGER
INTEGER
INTEGER

be parsed, enclosed in angle brackets (<>) such as<//GABR

ServiceStatus
Win32Status

INTEGER
INTEGER

Operation
Target

STRING
STRING

Parameters

STRING

[0100] The ODBC input data format accepts the folloWing

[0108] A ?le name, or a comma-separated list of ?le


names, including names that contain Wildcards, such
as LogFiles\W3SVC\ra*.ibl.

specifying the virtual site or site name Whose log ?les are to

IEGIO/W3SVC/1>, <//FARM/W3SVC/WWW.s5.com> or
<4>,<9> When referring to the local computer. If such a

source is speci?ed, the input source returns only those log


entries relative to the site speci?ed.
[0110] There are no options available for the BIN input
data format.

values in the FROM clause:

[0101] A complete speci?cation of the table from


Which the ?elds are to be extracted, in the folloWing
form:

[0102] table:<tablename>;DSN:<dsn>;username:
<username>;passWord:<pas sWord>

[0111] 7.0 URLSCAN

[0112] This input data format reads the URLScan log ?les
generated by the URLScan ?lter if it is installed on IIS.
Table 12 lists the URLScan ?eld names and corresponding

data types.

Dec. 16, 2004

US 2004/0254919 A1

[0125] 9.0 EVT


[0126] This input data format reads event information
from the WINDOWS Event Log, including System, Appli

TABLE 11
Data Type

Field

cation, Security, and custom event logs, as Well as from


LogFileName

MgROW

STRING
INTEGER

'
event log backup ?les (EVT log ?les). Table 14 lists
the EVT

Date

TIMESTAMP

input data format ?eld names and corresponding data types.

ClientIP

STRING

Comment

STRING

SiteInstance

INTEGER

Url

STRING

TABLE 14
Field

Data Type

EventLog

loWing values in the FROM clause:


.

[0114]

?le name or a comma-separated list of ?le

names, including names that contain W1ldcards, such


as URLSCan\*_10g_

[0115]

The URLSCAN command, to instruct log

parser
available
202URLScan
to retrieve
log and
?les.parse all the currently

[0116] The stdin command, used to pipe command


executions, such as type URLScan.log|LogParser
SELECT * from stdin-i:URLSCAN.

[0117]

There are no options available for the URLScan

STRING

ihTllEEggiMp

[0113] The URLScan input data format accepts the fol


Timewritten

TIMESTAMP

INTEGER

tID

EZZEtTYPe

INTEGER

EventTypeName
EventCategory

STRING
INTEGER

SourceName

STRING

Strings

STRING

glcgnputerName
Message

STRING

[0127] FROM clauses for the EVT input data format


accept a comma-separated list of names of EventLog (Sys
tem, Application, Security, or a custom event log) or EVT

input data format.

log ?les, optionally preceded by the name of the computer,

[0118] 8.0 HTTPERR

such as \\COMPUTER2\System. For example: SELECT

[0119] This input data format reads the IIS 6.0 HTTP error
log ?les. Table 13 lists the HTTPERR ?eld names and

\\COMPUTER2\System,

Message

corresponding data types.


TABLE 13
Field

Data Type

LogFileName
LogRoW

STRING
INTEGER

date
time

TIMESTAMP
TIMESTAMP

src-ip

STRING

src-port

INTEGER

dst-ip
dst-port

STRING
INTEGER

cs-version
cs-method
cs-url

STRING
STRING
STRING

sc-status

INTEGER

s-site

STRING

s-reason

STRING

FROM

System,

Application,

D :\MyEVTLogs\* .evt,

\COMPUTER5\Security.
[0128] The folloWing options are available for the EVT
input data format:
[0129] fullTeXt: Retrieves the full teXt of the event
log message; legal values are ON or OFF. The
default value is ON.

[0130] formatMsg: Formats the message, removing


carriage returns, line feeds, and eXtra spaces. Legal
values are ON or OFF. The default value is ON.

[0131] ignoreMsgErr: Ignores errors that occurred


While retrieving the full teXt of the event log mes
sage. Legal values are ON or OFF. The default value
is OFF. If these errors are not ignored and an error

occurs While retrieving the teXt of the message, the


entry itself is not returned. Conversely, if these errors
are ignored and an error occurs While retrieving the

teXt of the message, the entrys Message ?eld is

[0120] The HTTPERR input data format accepts the fol


loWing values in the FROM clause:
[0121]

A ?le name or a comma-separated list of ?le

names, including names that contain Wildcards, such

as HttpErr5*.log, HttpErr7*.log.
[0122] The HTTPERR command, to instruct log parser
202 to retrieve and parse all the currently available HTTP
error log ?les.

[0123] The stdin command, used to pipe command eXecu


tions, such as type HttpErrl.log|LogParser SELECT * from
stdin-i:HTTPERR.
[0124]

There are no options available for the HTTPERR

input data format.

returned as NULL.

[0132] fullEventCode: When this option is set to ON,


log parser 202 returns the full 32-bit value of the
EventID code. When set to OFF, log parser 202
returns the loWer 16-bit value of the code (as dis
played by the Event VieWer). The default value is
OFF.

[0133] resolveSIDs: Resolves all the retrieved SIDs


into fully speci?ed account names; legal values are
ON or OFF. The default value is OFF.

[0134]

10. TEXTWORD and TEXTLINE

[0135] These input data formats eXtract Words and full


lines from generic teXt ?les.

Dec. 16, 2004

US 2004/0254919 A1
10
[0136]

TEXTWORD: The Text ?eld of this input

default value is ON. When this option is set to OFF,

data format is represented by any single Word (sepa


rated by spaces) in the text ?le.
[0137]

TEXTLINE: The Text ?eld of this input data

format is represented by any single line (separated by

the ?elds are named Fieldl, Field2, and so on.

[0152] dtLines: Speci?es that the input source ?rst


reads the speci?ed number of lines from the ?les,
trying to detect the ?eld types. Specifying 0 disables
the search, and all the ?elds are treated as STRING
values. The default value is 10.

CRLF or CR) in the text ?le.


[0138]

TEXTWORD and TEXTLINE use the same ?eld

names and corresponding data types; listed in Table 15.

[0153] tsFormat: Speci?es the timestamp format used


by the TIMESTAMP ?elds in the ?le. You can

specify any timestamp format. The default value is

TABLE 15

yyyy-MM-dd hh:mm:ss.

Field

Data Type

LogFileName

STRING

Index
Text

INTEGER
STRING

[0154]

[0139] The TEXTWORD and TEXTLINE input data for


mats accept the following values in the FROM clause:

To see hoW the ?elds are detected by the CSV input

data format, type the folloWing at the command line: log

parser-h-i:CSV <from\-entity>. For example: logparser-h


i:CSV mycsv?le.txt
[0155] 12. W3C
[0156] This input data format reads W3C format log ?les,
Which are ?les not speci?c to IISthat contain special

[0140]

A ?le name or a comma-separated list of ?le

names, including names that contain Wildcards, such

as D:\Files\*.txt, D:\*.log.

[0141] The stdin command, used to pipe command


executions, such as type ?lel.txt|IJogParser
SELECT * from stdin-i:WORD.

headers and space-separated lists of values. For example,


WINDOWS Media Services, Personal FireWall, and
Exchange all Write log ?les in this format. W3C ?elds are
determined at run time, depending on the ?les and the

speci?ed options, Which are listed later in this section.

[0157] The W3C input data format accepts the folloWing


values in the FROM clause:

[0142] There are tWo options available for the TEXT


WORD and TEXTLINE input data formats.

[0158] A ?le name or a comma-separated list of ?le


names, including names that contain Wildcards, such

[0143] iCodepage: Speci?es the codepage in Which


the ?les are encoded; legal values are 1252, 0 (cur

rent system codepage), 1 (Unicode), 65001 (UTF


8), and so on. The default value is 0.

as D:\Files\*.log, D:\?le.log.

[0159] The stdin command, used to pipe command


executions, such as type extendl.log|LogParser
SELECT * from stdin-i:W3C.

[0144] recurse: Speci?es that the search recurses all


subfolders. Legal values are ON or OFF. The default
value is OFF.

[0145] 11. csv


[0146] This input data format reads CSV text ?les, Which
are text ?les that contain comma-separated lists of values.
CSV input data format ?elds are determined at run time,
depending on the ?les and the speci?ed options, Which are

listed beloW. The CSV input data format accepts the fol
loWing values in the FROM clause:

[0160] The folloWing options are available for the W3C


input data format:

[0161] iCodepage: Speci?es the codepage in Which


the ?les are encoded. Legal values are 1252, 0

(current system codepage), 1 (Unicode), 65001


(UTF-8), and so on. The default value is 0.

[0162] dtLines: Speci?es that the input source ?rst


reads the speci?ed number of lines from the ?les,
trying to detect the ?eld types. Specifying 0 disables

A ?le name or a comma-separated list of ?le

the search, and all the ?elds are treated as STRING


values. The default value is 10.

names, including names that contain Wildcards, such


as D:\Files\*.csv, D:\?le.csv.

[0163] dQuotes: Speci?es that the STRING values in


the ?le are enclosed in quotation marks (). Legal

[0147]

[0148] The stdin command, used to pipe command


executions, such as type log.csv|LogParser

values are ON or OFF. The default value is OFF.

[0149] The folloWing options are available for the


CSV input data format:

[0164] separator: Speci?es the character that is con


sidered as the separator betWeen ?elds. Legal values
are any single character enclosed betWeen apostro
phes, for example , or |, or the special strings tab

[0150] iCodepage: Speci?es the codepage in Which

ter). For example, the Exchange Tracking log ?les

SELECT * from stdin-i:CSV.

and space. The default value is " (a space charac

the ?les are encoded; legal values are 1252, 0 (cur

use a tab character as the separator betWeen the

rent system codepage), 1 (Unicode), 65001 (UTF

?elds.

8), and so on. The default value is 0.

[0165]

To see hoW ?elds are detected by the W3C input

[0151] headerRoW: Speci?es that the input source

data format, type the folloWing at the command line: log

treats the ?rst roW of every ?le as a comma-separated


list of ?eld names. Legal values are ON or OFF. The

parser-h-i:W3C<from_entity>. For example: logparser-h


i:W3C myW3c?le.txt.

Dec. 16, 2004

US 2004/0254919 A1

[0166]

[0180]

13. PS

[0167] This input source reads ?le information from the


speci?ed path, such as ?le siZe, creation time, and ?le

The folloWing options are available for the W3C

output target:
[0181] rtp: When printing to the screen, this option

attributes. The FS input data format is similar to an advanced


dir command. Table 16 lists the FS ?eld names and corre

speci?es the number of roWs to print before the user


is prompted to press a key to continue. If set to 1,

sponding data types.

the roWs print Without interruption. The default value


is 10.
TABLE 16

[0182] oCodepage: Speci?es the output codepage.

Field

Data Type

Path
Name
Size
Attributes
CreationTime
LastAccessTime
LastWriteTime
FileVersion

STRING
STRING
INTEGER
STRING
TIMESTAMP
TIMESTAMP
TIMESTAMP
STRING

Legal values are 1252, 0 (current system codepage),


1 (Unicode), 65001 (UTF-8), and so on. The default
value is 0.

[0183] odquotes: Speci?es that STRING values


should be enclosed in quotation marks (). Legal
values are ON or OFF. The default value is OFF.

[0184]

odirtime: Speci?es a string to Write to the

#Date header directive. The default behavior is to


Write the current date and time.

[0168] FROM clauses for the FS input source can accept


a path or a comma-separated list of paths, including paths
that contain Wildcards, such as D:\Files\*.txt, D:\*.*. The

[0185] ?lemode: Speci?es the action to perform


When the output ?le already exists. If you set the
value to 0, log parser 202 appends to the existing ?le.

following option is available for the FS input source:

If you set the value to 1, log parser 202 overWrites


the existing ?le. If you set the value to 2, log parser

recurse: Speci?es that the search recurses all subfolders.


Legal values are ON or OFF. The default value is ON.

202 does not Write to the ?le. The default value is 1:

[0169] Exemplary Log Parser Output


[0170] Log parser 202 supports the folloWing output tar

gets:

overWrite the existing ?le.

[0186] The folloWing command, for example, creates a

W3C-format log ?le (e.g., output ?le 208 of FIG. 2)


[0171]

W3C: This format sends results to a text ?le

that contains headers and values that are separated by


spaces.

[0172]

containing some ?elds from the Event Log:

[0187] logparser Select TO_DATE(TimeGenerated)


as date, TO_TIME(TimeGenerated) as time, Even
tID as event-id, EventType as event-type, Source

IIS: This format sends results to a text ?le With

Name as sourcename FROM System TO exevent

values separated by commas and spaces.


[0173]

.log-o:W3C.

SQL: This format sends results to a SQL table.

[0174] Comma-Separated-Value (CSV): This format


sends results to a text ?le. Values are separated by
commas and optional tab spaces.

[0188]

[0175] XML: This format sends results to an XML


formatted text ?le.

[0176]

#SoftWare: Log Parser


#Version: 1.0
#Date: 2002-06-21 18:26:10
#Fields: date time event-id event-type sourcename

Template: This format sends results to a text

?le formatted according to a user-speci?ed template.

[0177] Native: This format is intended for vieWing


results on screen.

W3C

[0178]

The ?rst lines of an exemplary generated Exevent

.log ?le are as folloWs:

The W3C output format Writes results to a generic

2002-04-17
2002-04-17
2002-04-17
2002-04-17

11:31:19
11:31:19
11:31:19
11:30:53

6008 1 EventLog
6009 4 EventLog
6005 4 EventLog
10 4 redbook

2002-04-17
2002-04-17
2002-04-17
2002-04-17

11:31:31
11:31:37
11:31:37
11:31:47

37 4 W32Time
1101 2 SNMP
1001 4 SNMP
35 4 W32Time

2002-04-17 11:32:23 7035 4 Service Control Manager

W3C-format text ?le. At the top of the text ?le are W3C
headers describing the ?elds. Field names are generated
from the SELECT clause or from the aliases assigned to
them. Values are separated With spaces.

IIS

[0179] When Writing the TO clause With the W3C output

[0189]

format, you can use a single ?le name, or you can use the

Microsoft IIS ?le format. The resulting text ?le contains a


list of values separated by a space and comma, With no

stdout command to print results directly to the screen. If you


use a Wildcard character
in the speci?ed ?le name, the

This output format Writes ?elds according to the

headers. When Writing the TO clause With the W3C output

Multiplex feature is enabled. The Multiplex feature converts

format, you can use a single ?le name, or you can use the

the ?rst ?elds in the SELECT clause and substitutes them for
the Wildcards in the ?le name generation. For more infor
mation on Multiplex, see Multiplex Feature later in this
document.

stdout command to print results directly to the screen. If you


use a Wildcard character
in the speci?ed ?le name, the

Multiplex feature is enabled. The Multiplex feature converts


the ?rst ?elds in the SELECT clause and substitutes them for

Dec. 16, 2004

US 2004/0254919 A1

the Wildcards in the ?le name generation. For more infor


mation on Multiplex, see Multiplex Feature later in this
document.

[0190]

The following options are available for the IIS

[0201] dsn: Name of an optional local DSN to use for


the connection.

[0202] createtable: If the target table does not exist,


and this parameter is set to ON, then log parser 202
creates a table, deriving the column types and names
from the SELECT clause according to the type
mapping above. The default value is OFF.

output target:
[0191] rtp: When printing to the screen, this option
speci?es the number of roWs to print before the user
is prompted to press a key to continue. If set to 1,

the roWs print Without interruption. The default value


is 10.

[0203] cleartable: Clears the existing table before


storing results. The default value is OFF.

[0204] ?xcolnames: Removes illegal characters from


column names for tables that log parser 202 creates.
The default value is ON.

[0192] oCodepage: Speci?es the output codepage.


Legal values are 1252, 0 (current system codepage),

[0205]

1 (Unicode), 65001 (UTF-8), and so on. The default


value is 0.

in a W3C log ?le to a SQL table:

The folloWing command exports some of the ?elds

[0193] ?lemode: Speci?es the action to perform


When the output ?le already exists. If you set the
value to 0, log parser 202 appends to the existing ?le.
If you set the value to 1, log parser 202 overWrites
the existing ?le. If you set the value to 2, log parser

uri-stem as UriStem,
cs-uri-query as UriQuery FROM ex000123.log TO TestTable

202 does not Write to the ?le. The default value is 1:

username:giuseppini

logparser Select TOiTIMESTAMP(date, time) as Timestamp, cs


o:SQL
server:GABRIEGISQL driver:SQL Server database:LogDB

passWord:xxx createtable:ON

overWrite the existing ?le.

SQL
[0194] This output format sends the results to a SQL table
using the ODBC Bulk Add command. If the SQL table

[0206] The resulting exemplary table contains the folloW


ing information:

already exists, the SELECT clause must match the SQL


table columns in type and number. In addition, the ?elds in
the SELECT clause must appear in the same order as the

Timestamp

SQL table columns. If the SQL table does not yet exist and

1/1/2002 12:00:01

/Default.htm

UriStem

UriQuery
<NULL>

the createtable option is speci?ed, log parser 202 creates the


table, deriving the column types and names from the

1/1/2002 12:00:03

/default.asp

PageID=4

1/1/2002 12:00:03

headergif

<NULL>

SELECT clause. Table 17 lists the type mapping for neW

SQL tables.
CSV
TABLE 17

[0207]

Log Parser Type

SQL Type

INTEGER
REAL
STRING
TIMESTAMP

int
real
varchar
datetime

[0195] The argument of the TO clause is the name of the


table. The folloWing options are available for the SQL output
format:

This format Writes results to a text ?le using the

comma-separated values format. After an optional header,


all values appear, separated by commas and optional spaces.
When creating the TO clause With the CSV output format,
you can use a single ?le name, or you can use the stdout

command to print results directly to the screen. If you use a

Wildcard character
in the speci?ed ?le name, the Mul
tiplex feature is enabled. The Multiplex feature converts the
?rst ?elds in the SELECT clause and substitutes them for the
Wildcards in the ?le name generation. For more information
on Multiplex, see Multiplex Feature later in this docu
ment.

[0196]

server: Speci?es the name of the server host

ing the database.


[0197] database: Speci?es the database name Where
the table resides.
[0198]

driver: Speci?es the name of the driver to use

during the ODBC operation. To specify SQL Server,


enclose the value in quotation marks, such as -driv
er:SQL Server.
[0199] username: User name to use When connecting
to the database.

[0208] The folloWing options are available for the CSV


output format:
[0209] headers: Writes a ?rst line containing the ?eld
names. The default value is ON.

[0210]

tabs: Writes a tab character after every comma

separator. The default value is ON.

[0211] tsformat: Speci?es the timestamp format to


use for TIMESTAMP values. The default value is

yyyy-MM-dd hh:mm:ss.

[0212] oCodepage: Speci?es the output codepage.


Legal values are 1252, 0 (current system codepage),

[0200] passWord: PassWord to use When connecting


to the database.

1 (Unicode), 65001 (UTF-8), and so on. The default


value is 0.

Dec. 16, 2004

US 2004/0254919 A1

[0213] ?lemode: Speci?es the action to perform


When the output ?le already exists. If you set the
value to 0, log parser 202 appends to the existing ?le.
If you set the value to 1, log parser 202 overWrites
the existing ?le. If you set the value to 2, log parser
202 does not Write to the ?le. The default value is 1:

overWrite the existing ?le.


[0214]

FIELD><FIELD
[0221]

TYPE=INTE

The folloWing options are available for the XML

output target:
[0222] structure: Speci?es the structure type of the
XML document. Legal values are 1, 2, 3 and 4. The
default value is 1.

The folloWing command creates a CSV ?le con

taining information about all the ?les larger than 500 KB in


the System32 folder:

NAME=BytesSent

[0223] rootname: Speci?es the name of the ROOT


element in the XML document. The default value is
ROOT.

C:\Winnt\system32\*.* TO ?les.csv WHERE Size>512000 i:FS o:CSV

[0224] roWname: Speci?es the name of the ROW


element in the XML document. The default value is
ROW.

[0215]

[0225] ?eldname: Speci?es the name of the FIELD


element in the XML document When the STRUC

logparser SELECT Name, Size, Attributes FROM

The resulting ?le is exempli?ed as folloWs:

TURE parameter has a value of 2 or 3. The

default is FIELD.
Name,
adminpakmsi,

Size,

Attributes

13135360,

.A _____ __

adpropdll,

740864,

.A _____ __

[0226] xslLink: Speci?es an optional link to an exter


nal XSL ?le to be referenced inside the XML docu
ment. The link is not speci?ed by default.

[0227] schemaType: Type of the inline schema speci

advapi32.dll,

546304,

.A _____ __

autochkexe,

573952,

.A _____ __

?cation. Legal values are 0 (none) and 1 (DTD). The

autoconv.exe,

587264,

.A _____ __

autofmt.exe,

566784,

.A _____ __

default value is 1.

[0228]

compact: Writes the XML document sup

pressing carriage return/line feed, and space charac


ters. The default value is OFF.

XML

[0216]

The XML output format is used to Write results to

[0229]

an XML ?le. The XML ?le is structured as a sequence of

ROW elements, each containing a sequence of FIELD


elements. The FIELD elements are Written in four different

text other than the ROW elements, suitable for being


concatenated With other documents. The default
value is ON. Notice that setting this value to OFF
generates a document not compliant to the XML

formats, depending on the value of the STRUCTURE

parameter:
[0217]

When the STRUCTURE parameter has a value of

speci?cations.

1, the FIELD elements have the same names as the ?elds

in the query result; for example, a ROW element looks like


this:
<ROW><UriStem>/default.htm</

[0230] oCodepage: Speci?es the output codepage.


Legal values are 1252, 0 (current system codepage),

UriStem><BytesSent>242</BytesSent></ROW>.
[0218]

1 (Unicode), 65001 (UTF-8), and so on. The default


value is 0.

When the STRUCTURE parameter has a value of

2, the FIELD elements have the same names as the ?elds

in the query result, and each element has a TYPE attribute

[0231] ?lemode: Speci?es the action to perform


When the output ?le already exists. If you set the
value to 0, log parser 202 appends to the existing ?le.

describing the data type. For example, a ROW element looks


like this: <ROW><UriStem TYPE=STRING>/de

fault.htm</UriStem><BytesSent

If you set the value to 1, log parser 202 overWrites


the existing ?le. If you set the value to 2, log parser

TYPE=INTE

GER>242</BytesSent></ROW>.
[0219] When the STRUCTURE parameter has a value of
3, the FIELD elements are named FIELD, and each
element has a NAME attribute describing the name of the

?eld; for example, a ROW element looks like this:

<ROW><FIELDNAME=UriStem>/default.htm</
FIELD><FIELD
NAME=BytesSent>242</FIELD></
ROW>.

[0220] When the STRUCTURE parameter has a value of


4, the FIELD elements are named FIELD, and each
element has a NAME attribute describing the name of the
?eld and a TYPE attribute describing the data type. For
example, a ROW element looks like this: <ROW><FIELD

NAME=UriStem

TYPE=STRING>/default.htm</

standAlone: Writes a fully-compliant XML

document With the <XML> header and every ROW


element embedded in a global ROOT element. Set
ting this value to OFF generates a document With no

202 does not Write to the ?le. The default value is 1:

overWrite the existing ?le.


[0232] The folloWing command Writes an XML document
containing the Url and BytesSent ?elds from an IIS W3C log
?le: logparser SELECT cs-uri-stem as Url, sc-bytes as

BytesSent from ex000805.log to Report.xml-o:XML-struc


ture:2,
[0233]

The resulting exemplary ?le appears as folloWs:

Dec. 16, 2004

US 2004/0254919 A1

-continued
<!ATI'LIST ROOT DATELCREATED CDATA #REQUIRED>
<!ATI'LIST ROOT CREATEDLBY CDATA #REQUIRED>

<!ELEMENT Url (#PCDATA)>


<!ATI'LIST Url TYPE CDATA #REQUIRED>

<!ELEMENT BytesSent (#PCDATA)>


<!ATI'LIST BytesSent TYPE CDATA #REQUIRED>

<!ELEMENT ROW (Url, BytesSent)>


<!ELEMENT ROOT (ROW*)>

]>
<ROOT DATEiCREATED=2002-11-07 22:04:54 CREATEDLBY=

Log Parser V2.0>


<ROW>
<Url TYPE=STRING>

[0240] In addition, one can include the optional TPL


HEADER and TPLFOOTER parameters to specify that a
header is Written at the beginning, and a footer is Written at
the end of the output ?le.

[0241]

2.0 Structured Format

[0242] In the structured format, the template ?le contains


<LPBODY> and </LPBODY> tags, Which enclose the text
that is output for each roW. Optional <LPHEADER> and
</LPHEADER> tags enclose header text. Any text outside
these tags is considered comment text and are ignored by
Log Parser. The BODY section can contain special %
?eldname % tags that are substituted at run time With the

/logparser
</Url>
<BytesSent TYPE=INTEGER>
3890

values of the speci?ed ?elds. At the end of the BODY


section are optional <LPFOOTER> and </LPFOOTER>
tags that enclose the footer text. The folloWing is a sample

structured format template ?le called mytemplate.txt:

</BytesSent>
</ROW>
<ROW>
<Url TYPE=STRING>

/logparser/chartquery.asp

<LPHEADER>This is my template. </LPHEADER>

</Url>
<BytesSent TYPE=INTEGER>

Some comment here.

taken% milliseconds to execute.


It Was requested at %time% oclock.

</BytesSent>

<LPBODY>The Url %csuristem%, requested by %cip%, took %time

</ROW>
<ROW>
<Url TYPE=STRING>

<LPFOOTER>End of report.
</LPFOOTER>

/logparser/chartit.asp
</Url>
<BytesSent TYPE=INTEGER>
0

[0243] To use this template, type the folloWing command:

</BytesSent>

LogParser SELECT * from extend1.log to out.txt-o:TPL

</ROW>
</ROOT>

tpl:mytemplate.txt. The resulting ?le contains the folloWing


information:
TPL
This is my template.

[0234] The template output target Writes results according

The Url /default.htm, requested by 192.141.56.132,

to a user-speci?ed template ?le. There are tWo different


formats in Which template ?les can be Written: raW format
and structured format.

took 24 milliseconds to execute.


It Was requested at 04:23:45 oclock.

[0235]

1. RaW Format

[0236]

In the raW format, the template ?le contains the text

that is output for each roW. The text can contain special %
?eldname % tags that are substituted at run time With the

values of the speci?ed ?elds. The folloWing is a sample raW

format template ?le called mytemplate.txt:

[0237] The Url % cs-uri-stem %, requested by % c-ip


%, took % time-taken % milliseconds to execute.

[0238]

It Was requested at % time % oclock.

[0239] To use the template, type the folloWing command:

The Url /mydocuments/index.html, requested by


192.141.56.133, took 134 milliseconds to execute.
It Was requested at 04:23:47 oclock.

End of report.

[0244] If one uses the TPLHEADER and TPLFOOTER


parameters to specify a header or footer ?le, these override

the header and footer text placed in the template. Note: In


this implementation, the log parser 202 assumes that the

character immediately folloWing the opening tag for a


section, such as <LPBODY>, belongs to that section

[0245]

The folloWing options are available for the TPL

output target:

LogParser SELECT * from extend1.log to out.txt-o:TPL

[0246] tpl: Speci?es the path to the template ?le.

tpl:mytemplate.txt. The resulting ?le contains the folloWing

[0247] tplheader: Speci?es the path to an optional

information:

header ?le.

[0248] tplfooter: Speci?es the path to an optional


footer ?le.
The Url /default.htm, requested by 192.141.56.132,
took 24 milliseconds to execute.
It Was requested at 04:23:45 oclock.

The Url /mydocuments/index.html, requested by


192.141.56.133, took 134 milliseconds to execute.
It Was requested at 04:23:47 oclock.

[0249] oCodepage: Speci?es the output codepage.


Legal values are 1252, 0 (current system codepage),
1 (Unicode), 65001 (UTF-8), and so on. The default
value is 0.

[0250] ?lemode: Speci?es the action to perform


When the output ?le already exists. If you set the

Dec. 16, 2004

US 2004/0254919 A1

value to 0, log parser 202 appends to the existing ?le.

[0261] An Exemplary Multiplex Feature

If you set the value to 1, log parser 202 overwrites


the existing ?le. If you set the value to 2, log parser
202 does not Write to the ?le. The default value is 1:

path automatically enable the Multiplex feature. Multiplex

overWrite the existing ?le.

[0262]

For most output targets, Wildcards in the target ?le

converts the ?rst ?elds in the SELECT statement to strings

and substitutes them for the Wildcards in the ?le path

[0251] 3.0 NAT


[0252] The Log Parser Native output format is intended to
shoW results on screen. If you Want to Write results to a ?le,
you can use a single ?le name as the argument of the TO

clause. Use the stdout command to print directly to the


screen. If no TO clause is speci?ed, log parser 202 prints to
the screen. The folloWing options are available for the NAT

output format:

generation. These ?elds are not output as results. For


example, if you Want to Write all the event log messages to

different ?les according to the event source, type the fol

loWing command:

[0263] logparser SELECT SourceName, Message


FROM System TO eventlogs\*.txt Where Event
TypeName=Error event-i:EVT-o:CSV The values
of the SourceName ?eld are substituted for the

[0253] rtp: When printing to the screen, this option

Wildcard character
in the output ?le name, and
the Message ?eld alone is output. The query results

speci?es the number of roWs to print before the user


is prompted to press a key to continue. If set to 1,

in formulation of ?les, containing the messages from


the system event log, look for example, as folloWs:

the roWs print Without interruption. The default value


is 10.

[0254]

headers: Writes a header line containing the

06/20/2002 05:07 PM

?eld names every time a neW screen group is printed.

06/20/2002 05:07 PM

The default value is ON.

[0255]

spacecol: Spaces all the ?elds in the same

screen group equally. The default value is ON.

06/20/2002
06/20/2002
06/20/2002
06/20/2002

05:07
05:07
05:07
05:07

PM
PM
PM
PM

06/20/2002 05:07 PM

[0256] ralign: When set to ON, the ?elds are right


aligned. The default value is OFF.

06/20/2002
06/20/2002
06/20/2002
06/20/2002

[0257]

06/20/2002 05:07 PM

colsep: Speci?es the character to use When

spacing the ?elds. Default value is a space.

[0258] oCodepage: Speci?es the output codepage.


Legal values are 1252, 0 (current system codepage),
1 (Unicode), 65001 (UTF-8), and so on. The default
value is 0.

[0259] ?lemode: Speci?es the action to perform


When the output ?le already exists. If you set the
value to 0, log parser 202 appends to the existing ?le.
If you set the value to 1, log parser 202 overWrites
the existing ?le. If you set the value to 2, log parser
202 does not Write to the ?le. The default value is 1:

overWrite the existing ?le.


[0260]

The folloWing command prints to the screen all the

URLs hit on your server running IIS, together With the error
response status code: logparser SELECT cs-uri-stem, sc
status FROM <1> WHERE sc-status>=400. The resulting

05:07
05:07
05:07
05:07

PM
PM
PM
PM

..

223,001
3,957
35,425
192

BROWSER.txt
Cdrom.txt
DCOM.txt
Dhcp.txt

2,078 EventLog.txt
292
9,826
13,113
765

IIS Con?gtxt
Kerberostxt
LsaSrv.txt
MRxSmb.txt

81 NetBTtxt

06/20/2002 05:07 PM

5,717 NETLOGON.txt

06/20/2002 05:07 PM

837
4,293
8,422
158
266
330
856
1,066
9,148
1,341

06/20/2002
06/20/2002
06/20/2002
06/20/2002
06/20/2002
06/20/2002
06/20/2002
06/20/2002
06/20/2002

05:07
05:07
05:07
05:07
05:07
05:07
05:07
05:07
05:07

PM
PM
PM
PM
PM
PM
PM
PM
PM

21 File(s)

321,164

nv4.txt

Servertxt
Service Control Managertxt
Setup.txt
SideBySide.txt
System Error.txt
TermDD.txt
TermServDevices.txt
W32Time.txt
W3SVC.txt

bytes

[0264] The folloWing example converts IIS binary log


?les, each of Which contain entries for all sites on a server,
to the W3C Extended log format structure. The result is

separate sets of ?les, each in a folder identi?ed by Site ID,


With ?les separated according to the date the requests Were
received.

screen lists, for example, the folloWing values:


logparser SELECT SiteID,TOiSTRING(DateTime,yyMMdd),
TOiDATE(DateTime) AS date, TOiTIME(DateTime) AS time, UriStem
AS cs-uri-stem FROM ra".ibl TO W3SVC*\ex*.log i:BIN o:W3C
cs-uri-stem

sc-status

/scripts/..L>>../Winnt/system32/cmd.exe
/scripts/..l../Winnt/system32/cmd.exe

404
404

/scripts/..%5c../Winnt/system32/cmd.exe
/scripts/..%5c../Winnt/system32/cmd.exe
/scripts/..%5c../Winnt/system32/cmd.exe
/scripts/..%2f../Winnt/system32/cmd.exe
/scripts/root.exe

404
404
404
404
404

/MSADC/root.exe

404

/c/Winnt/system32/cmd.exe
/d/Winnt/system32/cmd.exe

404
404

Press a key...

404

[0265] The ?rst tWo ?elds (SiteID and the log entry
timestamp formatted as yyMMdd) are substituted for the
tWo Wildcards in the target ?le name, and the folders and ?le
names are created accordingly. The result is similar to the

folloWing exemplary structure:

[0267] ex020618.log
[0268] ex020619.log
[0269] ex020620.log

Dec. 16, 2004

US 2004/0254919 A1
16

query automatically. Conversion mode is for con


verting one log ?le format to another.

[0271] ex020618.log
[0272] ex020620.log

[0286] Help: In Help mode, log parser 202 displays


information about hoW to use the tool.

[0273] W3SVC3\
Standard Mode

[0274] ex020618.log
[0275] ex020619.log
[0276] ex020621.log

[0287] In standard mode, you specify the input data format

[0277] Exemplary Log File Format Conversions


[0278] When using log parser 202 to convert one log ?le

and its parameters, the output format and its parameters, the
SQL query, and other global parameters. Standard mode is
the default. The folloWing example lists the syntax for
standard mode:

format to another, pay close attention to the order and names

of the ?elds in the input and output formats. Some output


formats, such as the IIS log format, have ?xed ?elds. When

converting to IIS log format, select ?elds from the input data
format that match the IIS format. For example, When con

verting a W3C Extended log ?le to IIS log format, select the
client IP address ?rst, the user name next, and so on.

[0279] In addition, you might Want to change the name of


the ?elds that you extract from the input data format. For
example, When Writing to an IIS W3C Extended format log
?le, log parser 202 retrieves the names to be Written in the
#Fields directive from the SELECT statement. If you
retrieve data from an IIS log format ?le, these names are not
the same as those used by the W3C Extended format, so use
the AS statement for every ?eld in order to get the correct

[0288] Table 18 lists the parameters used in a standard


mode query.
TABLE 18
Parameter

Description

i:<inputiformat>

IISW3C, NCSA, IIS, ODBC, BIN,


IISMSID, HITPERR, URLSCAN,
CSV, W3C, EVT, TEXTLINE,

?eld name.

[0280] Consider the folloWing built-in log parser 202

TEXTWORD, FS.

conversion query that converts IIS log format ?les to IIS

<inputiformatioptions>

Options speci?c to the input data

W3C Extended log format:

o:<outputiformat>

CSV, XML, NAT, W3C, IIS, SQL,

format selected.
TPL.

<outputiformatioptions>

Options speci?c to the


output format selected.
Maximum number of parse
errors before aborting. Default is 1

SELECT TOLDATE(TOLUTCTIME(TOiTIMESTAMP(Date,
Time))) as

(ignore all).

date, TOLTIME( TOLUTCTIME( TOiTIMESTAMP(Date,


Time))) as
time, ServiceInstance as s-sitename, HostName as s-computername,
ServerIP as s-ip, RequestType as cs-method, REPLACEiCHR(Target,

\u0009\u000a\u000d, +)

Ignore Warnings. Default is


OFF.

stats[ :ONOFF]

Dump statistics after


executing query. Default is

UserName as cs-username, UserIP as c-ip, StatusCode as sc-status,

ON.
Quiet mode. Quiet mode runs With

Win32StatusCode as sc-Win32-status, BytesSent


BytesReceived as cs-bytes, TimeTaken as time-taken

maxierrors = 1,

as cs-uri-stem, Parameters as cs-uri-query,

as

sc-bytes,

the following settings: No statistics,


iW = ON, and appropriate settings for
the NAT

[0281] Notice that the individual ?elds have been renamed


according to the IIS W3C Extended convention, so that the
output ?le is fully compliant With the IIS W3C Extended

output format; suitable for

exporting the
results to another application.
Default is OFF.

format. In addition, the date and time ?elds are converted

from local time, Which is used in the IIS log format, to UTC
time, Which is used in the IIS W3C Extended log format.

[0282] Exemplary Log Parser Command-Line Architec


ture

[0283]

Log parser 202 is available as a command-line tool,

LogParser.exe, Which has three operational modes:

[0284] Standard: In standard mode, you specify the


input data format, query, and output format, as Well
as other global parameters.

[0285] Conversion: In conversion mode, you specify


the input data format, output target, and an optional
WHERE clause, and log parser 202 generates a

[0289] In standard mode, if you do not specify an input


data format, log parser 202 tries to determine the format
based on the FROM clause. For example, if the FROM
clause is FROM extend*.log, then log parser 202 uses IIS
W3C Extended log ?le input data format because ?les in this
format are commonly named Extend1.log. The same applies
to the output target. If, for example, you specify ?le.csv as
the ?le to Which log parser 202 Writes results, then log parser
202 automatically uses the CSV output target.

[0290] If you do not specify the input data format and log
parser 202 cannot determine it, the TEXTLINE input source

is used. If you do not specify the output target and log parser
202 cannot determine it, the NAT output target is used.