Академический Документы
Профессиональный Документы
Культура Документы
February 2006
Executive Summary
National Instruments Citadel is a proprietary, real-time logging database designed primarily to
store single-point logging data. In contrast to relational databases, Citadel stores data in terms
data traces which are compressed and optimized for real-time logging and historical data
retrieval.
Citadel is an integral component of many National Instruments software products such as the
o
As a common data storage mechanism, Citadel allows these software products to easily exchange
test and measurement data. For example, a researcher could use DIAdem to analyze data
generated by the DSC Module or Lookout without having to modify, convert, or reprocess the
original data.
Citadel offers several major benefits that will improve productivity while saving time and
money. These benefits are:
Citadel is optimized for real-time logging and historical data retrieval. This results in
increased application performance while saving valuable system resources, especially in
large applications.
Citadel includes advanced data visualization and management components. You wont
have to develop custom data visualization or management tools for your application.
Citadel doesnt require any programming or prior knowledge of database systems to use.
This saves time and money when developing an application.
Citadel is inherently network-aware allowing you to share data seamlessly among team
members, or between data terminals.
Citadel is ready to use out-of-the box which greatly reduces start-up and system
familiarization time.
You do not need to be familiar with the contents of this white paper to be successful with Citadel.
However, if you are interested in learning about the technology behind Citadel or are planning a
large project using Citadel, you may want to take a few minutes to review this paper.
1/13
Table of Contents
Executive Summary_________________________________________________________________1
Citadel Database Structure___________________________________________________________2
Supported Data Types______________________________________________________________________3
Compression of Numeric Data________________________________________________________________3
Traces and Subtraces_______________________________________________________________________4
VI Logger Data____________________________________________________________________________6
Database Files____________________________________________________________________________7
Database Size Considerations________________________________________________________________8
Historical Alarm and Event Data______________________________________________________________8
Citadel Operations__________________________________________________________________9
Installing Citadel__________________________________________________________________________9
Creating, Attaching, or Detaching a Database____________________________________________________9
Archiving Data___________________________________________________________________________10
Backing Up and Restoring a Database_________________________________________________________10
Retrieving Data from a Database_____________________________________________________________11
Inserting Data into a Database_______________________________________________________________12
Local Databases
Network Databases
Figure 1: Citadel Database Layout
2/13
Notice that the Historical Data Viewer also gives you access to multiple databases on the same
computer and provides access to remote databases.
Besides traces views, Citadel also includes Waveform hierarchies which contain data logged
from VI Logger, and dataset hierarchies which are created using the DataSet Marking I/O server
in the DSC Module.
Analog
Analog values are compressed based on the specified logging resolution
Integer &Bit-array
Discrete are not compressed when logged and are returned as an I32 typed values
String
Citadel supports three string types; LabVIEW style string, null terminated strings,
and Unicode strings
Variant
Citadel stores raw variant data, but optimizes storage by only storing the metadata
associated with a variant once instead of with each logged value.
Waveform
VI Logger and DIAdem are the only two NI software products that can interact
with waveform data
to Citadel. Once in Citadel, the values are converted to deltas and compressed using the
specified logging resolution. The example shown in table 1 uses a logging resolution of 0.1
which means all values retrieved from the database will be within 0.1 of the actual recorded
value.
Citadel uses a complex compression algorithm to store data, but this compression can be
approximated using the following equation.
Equation 1: Simplified Citadel Delta Calculation
y k n 1
n1 n
resolution
The logging resolution is a key parameter in how Citadel compresses numeric data. The logging
resolution can be any real number but should be chosen in concert with the logging deadband.
For instance, if you specified a logging resolution of 10 units and a logging deadband of 1 unit,
Citadel would log any change of at least 1 unit, but due to compression within the Citadel
database any value change under 10 could not be discerned from data retrieved from the
database. To prevent this from happening you should always specify a logging resolution less
than or equal to the logging deadband. You should also specify the logging deadband as a power
of ten unless you have an application-specific reason to do otherwise. Specifying a logging
resolution of zero (0) will always log data with full resolution.
For reasons explained later in this white paper, the number of values in a run will always be less
than 100, i.e. n will always be less than or equal to 100.
4/13
Data type indicates the most generic type associated with a trace. The most generic type
indicates what sort of reader should be used to retrieve the majority of the traces contents.
Types from most to least generic are:
The startTime and endTime properties of the trace indicate the actual start and end times of the
trace in Universal Coordinated Time (UTC) format. UTC avoids all conflicts associated with
daylight savings time or varying time zones. Although Citadel stores timestamps in UTC, the
times displayed in the Hyperbrowser are shown in the local time zone. If you expand the value
column you will see that the current time zone name is appended to the end of the time string.
The lifespan property indicates how long trace data will remain in the Citadel database before
Citadel reuses storage space taken up by old trace data. For instance, if a trace were configured
with a lifespan of 10 days, only the 10 most recent days worth of data would remain in the
database. After 10 days the oldest data would be overwritten. The lifespan of a trace is defined
by the application that writes data to the database and can only be configured on a per-process
basis, not a per-trace basis. A lifespan of 0 indicates an infinite lifespan.
The data lifespan is not a contract. Setting a lifespan indicates signals Citadel that data older
than the lifespan can be deleted in order to save space, but the database does not guarantee that
expired data will be deleted on any schedule. Lifespannig only occurs when the process that
originally logged the data is actively logging new data to the database. For instance, if you
5/13
copied data to a CD and, 15 years later, reattached your database to retrieved data, the data would
not suddenly disappear due to the lifespanner.
The pages property indicates how many 4kb pages the traces is using and provides a rough
estimate of how much disk space is used by the trace.
The subtraces property indicates how many subtraces a trace is composed of.
A subtrace is an arbitrary data stream containing a single type of data, plus associated meta-data.
Subtraces are indexed by time, making it possible to seek to a time position within a subtrace
without reading through the entire stream. Because data points are indexed by time, all data in a
subtrace data stream must have nondecreasing timestamps. If a back-in-time event occurs, or the
type of the data changes, Citadel must create a new subtrace. Changing the type of the data
means changing the base data type (double, bool, etc.), or changing the format with which data is
serialized to disk (e.g. changing the compression algorithm by changing the value or time
resolution, changing the type of Variable being logged to a Variant subtrace, or even changing the
attributes that are set on the Variant. Setting attributes on Variants logged to Citadel is not
recommended.)
Though Citadel provides no direct interface to work with individual subtraces its important to
understand subtraces because they can have a significant impact on database performance. In
particular, if a trace contains many subtraces it will perform less well than a trace with fewer
subtraces. In typical use cases Citadel will not need to create multiple subtraces.
If any local Citadel client, reader, writer, or the Citadel service fails to shut down cleanly while
accessing a database, the database will be re-indexed on restart. When a database is re-indexed,
we do not know if a subtrace may have been somehow damaged by the failure that triggered the
re-indexing, so all existing subtraces are closed permanently, and new data will be logged into a
fresh subtrace.
The following events can trigger the creation of a new subtrace:
Subtraces are made of data runs of n = 100 points. This applies mainly to analog traces where
data is compressed based on delta values and the logging resolution. After every 100 points an
absolute measurement value is logged which resets the compression algorithm. This prevents
accumulation of error and reinforces data integrity within the database.
VI Logger Data
VI Logger stores waveform data to Citadel in groups of runs, which belong to VI Logger tasks.
Each run corresponds to a single acquisition of data. A VI Logger subtrace consist of between
one and three subtraces, a waveform subtrace containing data from hardware, a waveform
subtrace containing calculated data generated by the VI Logger application, and a subtrace
6/13
containing VI Logger event data. VI Logger subtraces contain raw waveform data, which are
essentially arrays of uncompressed values. Waveform subtraces do not contain serialized
timestamps because the waveform timestamp can be calculated from its start time and deltaT
attributes. VI Logger uses uncompressed subtraces to maximize throughput to disk, at the cost of
greatly increased disk and network usage. The VI Logger event subtrace uses the same
value/timestamp format as single-point traces to log arbitrary-length boolean arrays without
compression.
Database Files
Citadel databases are stored in a special set of files on your hard drive. A Citadel database
typically resides in a folder unique that that database. Though not required, you should avoid
placing non-database related files a folder that contains a Citadel database. You can only have
one Citadel database in a given folder. A typical database is comprised of a set of files similar to
those presented in figure 2.
The number of CDPG and CDIB files will vary depending on the amount of data in a database.
The nodetree.*, pid.cdih, and stridm.cdin files contain important information about the structure
of the database. The mssql.* files contain historical alarm information and are automatically
created the first time alarm or event data is written to the database.
Important: You should never modify, move, or delete a database file while the database is
attached. Doing so will result in a database corruption. If you modify or delete a database file
while the database is detached you may not be able to reattach the database and could loose
some or all of the data in the database. If you move or copy a detached database, be sure to
move/copy all database files.
The CDPG files contain trace data. Citadel stores data in a compressed format therefore it is not
possible to read and extract data from these files directly. You must use the Citadel API in
LabVIEW DSC or the HDV to access trace data. See the Citadel Operations section for more
information about retrieving data from a Citadel database.
Each of the 1,024 Kb CDPG files contains a set of 4 Kb pages. Each page contains data for a
single subtrace. Citadel will attempt to use all 4 Kb of space in a page before opening a new
7/13
page. If a subtrace is terminated or a new subtrace is started before the end of a page the
remainder of the space in the page will go unused. Citadel will only reclaim the remaining
unused space if all trace data in that page is removed or deleted.
Citadel database performance will degrade slightly as the database grows. This is due
primarily to the physical limitations of hard disks and limitations inherent in the NTFS
and FAT32 file systems.
Archiving large databases takes significantly longer than archiving smaller databases
Database integrity is not related to database size
Based on experience and interactions with a variety of users, a large database is typically in the
range of 5-10 GB, a medium-sized database is 2-5 GB, and a small database is less than 2GB.
In many applications, database life spans range from 1 to 15 years.
Properly managing database size will greatly improve your ability to maintain the database over
its expected life time. The following tips will help you plan a healthy database system:
8/13
Citadel Operations
Installing Citadel
The Citadel database component is automatically installed with the following National
Instruments Software:
o
Citadel is not an independent software component and cannot be installed as such. If you need to
use Citadel on a remote terminal, the recommended option is to purchase a LabVIEW DSC
Module Run time license for that terminal. The LabVIEW DSC Module Run time includes the
Citadel database drivers.
The following other components are also installed with Citadel,
NI Logos (NI-PSP),
Microsoft SQL Server 2000 Desktop Engine (MSDE 2000), and
If you intend to use the Historical Data Viewer component to view Citadel trace data you must
also install Measurement and Automation Explorer (MAX). MAX is automatically installed
with most National Instruments software.
9/13
database has been detached; this is the only safe state in which a user should manually move or
delete database files. You must detach a database before you manually copy the database files or
write them to permanent storage. If you fail to detach a database before manually copying
database files it is possible that you will not get an accurate snap-shot of database files and the
copied database may be corrupt when you try to attach it.
When detaching a database you have the option to delete the database files. This option is useful
if you no longer need the data in the database. Only local databases can be detached or deleted
from a system. Citadel does not allow these operations for networked databases.
Archiving Data
Archiving data allows you to copy trace data from one location to another. You can perform
archives of trace data within a database or to another local or remote database.
An archive operation requires the user to specify four pieces of information: the sources trace(s),
the destination database, and the start and end times for the archive operation. Citadel will not
archive data that has already been archived for a particular trace. For example, if you perform an
archive operation for a time range from 1/1/2005 to now on a daily basis, Citadel will archive
the initial range of data once, then it will only archive the latest days worth of data on
subsequent archive requests.
The lifespan of all archived trace data is automatically set to 0, or infinite. Unless the lifespan
property is reset by logging data to the archived trace, the data will not be affected by a lifespan.
Citadel archives data by physically copying the trace pages from one database to another. Since
Citadel must reconstruct a database page-by-page, performing an archive operation will take
significantly longer than it would to copying a raw file.
Two common use cases for archiving include trace merging and destructive archiving. Merging
allows you to take information from one trace and merge it into a trace with the same name. For
example, assume that, for redundancy purposes, you have two computers, A & B, logging the
same data source to local database. All of your client applications reference the database
computer A when the retrieve data. If computer A goes down, you need a way to merge the
redundant data from computer B into the same trace on computer A. This can be done using the
following procedure:
1. Create a temporary database on computer A, then
2. Archive the backup trace from computer B to this temporary database on computer A.
The trace will have the same name and path as it did on computer B.
3. Rename the archived trace in the temporary database on computer A to have the same
computer name as computer A
4. Archive the data from the temporary database to the master database on computer A.
This will create a new subtrace in the original trace that contains the redundant data.
10/13
Destructive archiving is used to move data from one database to another. This is a common
process when backing up older data to a more permanent or off-line storage database.
11/13
Trace views in the HDV can be configured to show multiple overlapping traces and can be
configured in trace groups.
The LabVIEW DSC Module is the most efficient way to programmatically retrieve data from
Citadel. The DSC module includes an advanced API for retrieving historical trace and
alarm/event data, which can be requested in interpolated or raw form. Raw data simply contains
the actual values recorded in the database and the associated time stamps associated with those
values. See Table 1 for an overview of raw data. A request for interpolated data requires the user
to specify a time range and time interval. The returned data will be an interpolated
approximation of what trace values were at a given time. The user can specify either linear or
stair-step approximation when requesting interpolated data.
Finally, National Instruments DIAdem provides a direct interface to Citadel which allows users
to perform advanced analysis and reporting directly on Citadel data. Visit ni.com/diadem for
more information about this powerful analysis package.
The Citadel database also includes an ODBC driver, which enables you to directly retrieve data
from 3rd party applications. The Citadel 5 ODBC driver is compliant with SQL 92 and ODBC
2.5 standards. All clients that follow these standards can retrieve data from a Citadel 5 database.
12/13
ADO clients can now use the ODBC driver through the Microsoft OLE DB Provider for ODBC
Drivers to access the Citadel 5 database.
The LabVIEW DSC Module 8 contains a set of VI functions that can used to manually insert
data into a Citadel trace. Data can be inserted point-by-point with either a user specified or
server generated time-stamp. Numeric, logical, string, bit-array, or variant data can all be written
using this method. Benchmarks have demonstrated that single point data can be written to a
Citadel trace at approximately 80k updates per second from memory. Once written to the
database, the data is immediately available to all local or remote Citadel clients.
Security
When planning a networked data logging system it is important to consider that Citadel does not
contain any built-in security features. Data must be secured by placing all PCs on the data
logging network on an isolated sub-net or an isolated network. Otherwise, any user with access
to Citadel on the network will be able to view, modify, or copy potentially sensitive Citadel data.
All Citadel terminals should also be password protected to prevent unauthorized access to data.
13/13