ARCHIVING

Introduction

T24 Archiving has been designed to reclaim space by moving historic data that is no longer
required. Archiving examines files for records to be archived, writes the selected records (and any
associated records) to an archive file (which should reside on a separate disk, or possibly a separate
machine if accessible via a network), and then deletes the records. It is also possible, if so required,
to simply delete the data without archiving it.

File Layout

For each of the sets of files for which archiving is to be run, there is a record present on the
ARCHIVE file. There are three distinct sections of the record. The first contains the main data of the
record and is used to specify such things as the Purge Date, Retention Period, whether data is to be
removed or archived, and if so where the archived data is to be stored.

The second fields (ARC.FILENAME - MODULUS) are related multi-value fields, which describe the
archive files, which will be created

The third section of fields (COMPANY.RUN.IN - TIME.ENDED) automatically maintains a history of

each time archiving was run.

Figure 1 Archive Record

The Archiving Process

Timing of the Archiving process

It is recommended that the archiving process be run after the Close of Business has been completed,
but before users are allowed to sign on. However, since archiving removes data that is no longer
used, it can be run whilst the system is being used.

Compiling/Cataloguing programs

If it is thought that the data which is going to be archived may be needed in the future for reporting
purposes, then before starting archiving, ensure that the database has programs catalogued locally.
The database can then be restored at a later date and reports run using the data. To check that the
programs are catalogued locally, look at field 11 (CATALOG) on the SYSTEM record on the SPF. This
field will be GLOBAL or null. If it is null, the cataloguing option will be LOCAL. However, care
should be taken if relying on this field since, if the option on the SPF was changed from GLOBAL to
LOCAL for example, only programs catalogued since the change was made will be catalogued
locally. Therefore, all VOC entries for programs should be examined to determine the cataloguing
option.

If programs are not catalogued locally, then, if the database is restored the programs will need to be
catalogued afterwards.

The system on which archiving is to be run should be fully backed up. It is important to note that
archiving is a one-way process: data cannot be restored afterwards, except by fully restoring the
database back to the point of pre-archiving.

Backups

Back up the area as follows:

From UNIX, change to the directory that is being backed up. For example, if the area being backed
up is bnk and the directory structure is /usr/GLOBUS/bnk/bnk.run then type

$cd /usr/GLOBUS
$find bnk -print -depth | cpio -ovcB > /dev/zzz

Where zzz is the name of your backup device.

From UNIX verify the tape as follows:

$cpio -iBct < /dev/zzz

Where zzz is the name of your tape device.

Running Archiving

Archiving is run on one set of files at a time. There is a separate record for each set of files to be
archived, the ids of which can be found in section 0 Files.

Before running archiving, it should be decided whether data should be archived and, if so, where
the archived files should reside. Enter this information onto the ARCHIVE record for the set of files
to be archived. If data is to be archived, files will be created with a $ARC suffix in the archive
directory. For example, if special entry files are to be archived, files will be created as
RE.CONSOL.SPEC.ENTRY$ARC and RE.CONSOL.SPEC.ENT.KEY$ARC.

The layout of the $ARC files will be the same as the live files, with concat files being recreated as
necessary. By default, the $ARC files will be created with the same type and modulus as the
corresponding live files. However, if, for example, only half the data is expected to be archived, it
may be desirable to create the $ARC file with a smaller modulus than the live file. This
information should be entered on the ARCHIVE record.

N.B. The file sizes below are examples only and should not be used. For more information on sizing
files refer to the File Maintenance User Guide.

Figure 2 Archive Record

NOTE: Although the file will exist in the area specified in $ARC.PATHNAME, the file is
temporarily created in the run account, before being moved to the ARC directory. Therefore, there
must be enough space in the run account to create the largest of the files (each file is created, moved
and deleted before the next file is created).

The retention period should then be determined. This can be done either by entering a specific
purge date (which must be the first of a month) or by using the retention period. If todays date is
13/06/03 and a retention period of 3 months is specified, three months is calculated from the
beginning of the month. Therefore, any records from before 1/03/03 will be archived.

The CHECK.NO.OF.RECS field can be amended if required. This field determines the number of
records to be checked before checking whether the stop indicator has been set. It also determines
how often messages will be displayed showing the status of archiving (number of records
processed, number of records deleted). As the size of the files to be archived differs and the
processing times for different archiving jobs vary, this field will be different on each ARCHIVE
record.

Once the data is correct, verify it (or press F5/commit if already in verify mode). An override
message will be displayed showing the date, which will be used for archiving data. At this point
archiving can be terminated without any updates taking place.

To continue, type Y and <RETURN> and archiving will start. Messages will be displayed showing
the status of the archiving process, including the number of records processed and deleted.
Messages are displayed depending on the CHECK.NO.RECS field on the ARCHIVE record.
Information regarding which files the number of records processed and deleted are displayed for is
given in section 0 Files.

To halt the process, simply input Y into the STOP.INDICATOR field on the relevant ARCHIVE
record. This needs to be performed on a separate GLOBUS login, as the login session running
archiving cannot be used. Once the data has been committed (the F5 key hit) the archiving process
will come to a controlled halt after the next block of records has been processed (this block size is
controlled by the field CHECK.NO.RECORDS).

Post Archiving Actions

Once the archiving process is complete the file sizes of all of the files involved (both original and
$ARC where applicable) should be reviewed to take into account the new number of records in the
relevant files.

For more information on sizing files refer to the File Maintenance User Guide.

Reports

A COMO will be produced called ARC.xxxxx where xxxxx is the name of the ARCHIVE record,
e.g. ARC.SPECIAL.ENTRY.

Accessing the archived information

The archived information will be written to files with the suffix $ARC. The $ARC files use the
same layout as the files from which they have been created e.g. the layout of
RE.CONSOL.SPEC.ENTRY$ARC is the same as RE.CONSOL.SPEC.ENTRY.

Creating enquiries to view archived information

The archived files can be viewed using the T24 ENQUIRY utility, by creating an enquiry to view
the archived file. This enquiry can be a copy of an enquiry on the original file that was archived,
setting the FILE.NAME to the name of the archived file. For example, an enquiry to view the
TELLER$ARC file could be based on the TELLER enquiry, changing the FILE.NAME to
TELLER$ARC.

Figure 3 - Creating an Enquiry to view archived information

Viewing archived records

To enable you to drilldown to view the archived records, the following fields should be added to the
archive enquiry:

Figure 4 - Viewing archived records

The variable ARC.FILE.NAME should be assigned to the name of the file, e.g. for the teller archive
enquiry, ARC.FILE.NAME should be assigned to TELLER (enter TELLER in the OPERATION
field).

DEFAULT.ARCHIVE.VIEW is a generic enquiry that is supplied in T24, to enable viewing

(drilldown) of the archived records.

Files

CATEG.ENTRY - Category entries

Category entries are processed first, followed by the consol profit file. For category entries,
CATEG.MONTH is the driving force; for consol profit, RE.CONSOL.PROFIT is the driving force.
As the category entries are used for rebuilding the P&L, entries for the current financial year cannot
be archived. Therefore, the purge date is validated to ensure that it is before the last financial year-
end.

The CATEG.MONTH file is selected in id order, i.e. by category, by date. If the date in the key is
older than the purge date, all the category entries listed on the record will be removed from the
category entry file, CATEG.ENTRY and the record removed from CATEG.MONTH. If there are
more than 198 category entries for one category in a month, subsequent categ month records will
have been created. Archiving will process all categ month records for one category for one month
before checking the stop flag. Therefore, if there are three categ month records for one category for
one month, all three records will be archived before the stop flag is checked.

Once the categ month file has been processed, the RE.CONSOL.PROFIT file is processed,
(providing archiving has not been stopped). For each record on the consol profit file, the date in the
id is checked against the purge date. If the date is before the purge date, the record is archived.

The CHECK.NO.RECS and records processed refer to both the CATEG.MONTH file and the
RE.CONSOL.PROFIT file. The records deleted refer to both the CATEG.ENTRY file and the
RE.CONSOL.PROFIT file. Therefore the records processed will be a total of CATEG.MONTH
records and RE.CONSOL.PROFIT records and the records deleted will be a total of
CATEG.ENTRY records and RE.CONSOL.PROFIT records. As for each record on
CATEG.MONTH there could be up to 200 records to archive from CATEG.ENTRY, though a
figure of 100 is recommended as this allows changes to be made on-line if necessary. Where this is
not possible the total CHECK.NO.RECS should be set fairly low.

The ARCHIVE record for category entries is CATEG.ENTRY.

DELIVERY - Delivery

Outward delivery messages are processed first, followed by inward messages. For outward
messages, the outward delivery archive file, DE.O.HEADER.ARCH is the driving force; for inward
messages, the inward delivery archive file, DE.I.HEADER.ARCH is the driving force.

For each header record on the header arch file (provided the date in the key of the record is before
the purge date and the record is not still in the live file (DE.O.HEADER)), the message records for
each copy of the message are removed from the history file, DE.O.HISTORY; the key is removed
from the concat file, DE.O.HISTORY.QUEUE and the record removed from the header arch file,
DE.O.HEADER.ARCH.

Processing is then repeated for the inward delivery archive file.

The CHECK.NO.RECS, records processed and records deleted all refer to the DE.O.HEADER.ARCH
and DE.I.HEADER.ARCH files. Processing for each record is fairly simple so the CHECK.NO.RECS
should be set fairly high. A figure of 1,000 is suggested which can be changed on-line if necessary.
NOTE: The archiving process for delivery starts with a select of the DE.O.HEADER.ARCH file.
This file is large and therefore this selection could take some time. After all outward messages have
been processed, the inward delivery arch file, DE.I.HEADER.ARCH file will be selected.

NOTE: The archiving of delivery only works on completed messages on the delivery archive files,
i.e. records that have been copied to delivery archive files and removed from the live delivery files.
This is done in the delivery end of period programs, DE.MM.O.END.OF.PERIOD and
DE.MM.I.END.OF.PERIOD. Therefore, the archiving of delivery will only remove records
provided the delivery end of period has been run.

Records archived from the delivery header archive files will be written to DE.O.HEADER$ARC
and DE.I.HEADER$ARC.

The ARCHIVE record for delivery is DELIVERY.

FOREX Foreign Exchange

The FOREX history file is processed to produce a list of contract ids. For each contract, if there is
no corresponding record on the live FOREX file and the date from the DATE.TIME of the last history
record is before the PURGE.DATE/RETENTION.PERIOD from the ARCHIVE record, all the history
records for the contract are archived.

The ARCHIVE record for the foreign exchange history file is FOREX.

FUNDS.TRANSFER Funds Transfer

The FUNDS.TRANSFER history file is processed to produce a list of contract ids. For each
contract, if there is no corresponding record on the live FUNDS.TRANSFER file and the date from
the DATE.TIME of the last history record is before the PURGE.DATE/RETENTION.PERIOD from the
ARCHIVE record, all the history records for the contract are archived.

The ARCHIVE record for the funds transfer history file is FUNDS.TRANSFER.

MM.MONEY.MARKET Money Market

The MM.MONEY.MARKET history file is processed to produce a list of contract ids. For each
contract, if there is no corresponding record on the live MM.MONEY.MARKET file and the date
from the MATURITY.DATE of the last history record is before the PURGE.DATE/RETENTION.PERIOD
from the ARCHIVE record, all history records for the contract are archived.

Money Market contracts can have many other associated files, which require tidying up to avoid
data corruption. Therefore the following files: MM.PAYMENT.ENTRY$HIS,
MM.RECEIPT.ENTRY$HIS, LMM.ACCOUNT.BALANCES.HIST, LMM.HISTORY.LIST and
LMM.SCHEDULES.PAST.HIST which could contain references to the Money Market contract on
the history file are checked and archived as necessary.

The ARCHIVE record for the money market history file is MM.MONEY.MARKET.

SPECIAL.ENTRY - Special entry files

RE.CONSOL.SPEC.ENTRY is the driving file. When archiving is run on special entry files,
records older than the specified date will be removed from RE.CONSOL.SPEC.ENTRY and the
reference to the record will be removed from the concat file, RE.CONSOL.SPEC.ENT.KEY.

The CHECK.NO.RECS, records processed and records deleted all refer to the special entry file,
RE.CONSOL.SPEC.ENTRY. As the processing for each record is fairly quick and simple, the
CHECK.NO.RECS can be set higher for this ARCHIVE record than for others. A figure of 10,000 is
suggested which can be changed on-line if necessary.

NOTE: The archiving process for special entries starts with a select of the special entry file. As
this file can be very large, this select could take some considerable length of time.

The ARCHIVE record for special entries is SPECIAL.ENTRY.

STATEMENT - Statements

Frequency 1 statements are processed first, followed by frequency 2 statements. For frequency 1
statements, the ACCT.STMT.PRINT file is the driving force; for frequency 2 statements, the
ACCT.STMT2.PRINT file is the driving force.

For each account record on the ACCT.STMT.PRINT file, all statement ids are processed except the
last one, so that one statement is always left for frequency 1 and frequency 2 statements. However,
as soon as there is an outstanding entry on a statement, no more statements are processed for that
account.

For each statement id, the corresponding record is read from the STMT.PRINTED file. Each entry
on the statement is checked to ensure that it is does not appear on the most recent frequency 2
statement (which is not deleted) and that it is not on the ACCT.STMT.ENTRY or
ACCT.STMT2.ENTRY record which would include all entries for this account which still have to
be printed. Providing the entry has been printed, the record is then read from the STMT.ENTRY
file, to ensure that the booking date is before the purge date and that the value and exposure dates
are not in the future. Finally, AC.STMT.HANDOFF file is checked to ensure the record is not still
present on it. Providing all entries on a statement match these criteria:

The entries are deleted from the STMT.ENTRY file.

The statement printed record is deleted from STMT.PRINTED file.
The statement details are removed from the STMT.PRINTED file.
The statement record is removed from the AC.STMT.HANDOFF$HIS file and the statement
details are removed from the ACCT.STMT.PRINT file.

The processing is then repeated for frequency 2 statements.

The CHECK.NO.RECS and records processed refer to the ACCT.STMT.PRINT and

ACCT.STMT2.PRINT files. The records deleted refer to the statement entry file, STMT.ENTRY.
As the processing for each record on ACCT.STMT.PRINT is very complex and therefore slow,
since all entries for all statements for an account will be checked, the CHECK.NO.RECS should be set
relatively low. A figure of 100 is suggested which can be changed on-line if necessary.

NOTE: The archiving process for statements starts with a select of the ACCT.STMT.PRINT file.
This file contains the same number of records as the ACCOUNT file and therefore this selection
should not take too long. After all frequency 1 statements have been processed, the
ACCT.STMT2.PRINT file will be selected.
The ARCHIVE record for statements is STATEMENT.

SEC.TRADE - Security Transactions

Initially, the live SEC.TRADE file is processed, followed by the SEC.TRADE history file.

The live SEC.TRADE file is selected. This can be a large file, and the selection process is therefore
likely to take some time. Each record within the date selection is deleted from the file, providing
that it does not exist on the unauthorised file.

The deleted records are archived, if requested.

The SEC.TRADE History file is then selected. These records are deleted if they have no
corresponding record on the live file.

The History records are NOT archived.

The CHECK.NO.OF.RECS, RECS.DELETED and RECS.PROCESSED fields refer to the SEC.TRADE live
file and the SEC.TRADE$HIS history file. Disregarding the history file, which is likely to have a
minimal record count, the Sec Trade processing is fairly straightforward. An appropriate figure to
be used in the CHECK.NO.OF.RECS field is the default of 1,000.

The ARCHIVE record for securities is SEC.TRADE.

TELLER

The TELLER history file is processed to produce a list of contract ids. For each contract, if there is
no corresponding record on the live TELLER file and the date from the MATURITY.DATE of the last
history record is before the PURGE.DATE/RETENTION.PERIOD from the ARCHIVE record, all
history records for the contract are archived.

The ARCHIVE record for the teller history file is TELLER.