ILM Planning

Guide
Optimizing your storage costs for Oracle Utilities Application Framework
products

WHITE PAPER / JANUARY 2019

DISCLAIMER
The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.

2 W HITE PAPER / ILM Planning Guide

Table of Contents
Introduction .................................................................................................. 5

What Is Information Lifecycle Management? ............................................... 5

Why is Information Lifecycle Management Important? ................................................................ 5

Deletion vs Information Lifecycle Management ........................................................................... 6

ILM Concepts .............................................................................................. 7

Lifecycles ..................................................................................................................................... 7

Data Types (Scope of ILM) .......................................................................................................... 8

Data Activity................................................................................................................................. 9

Business Versus Technical Aspects ............................................................................................ 9

Simple versus Complex Solutions ..............................................................................................10

Data Management Solution Overview ....................................................... 11

Features and Functions ..............................................................................................................11

ILM Facilities within the Product .................................................................................................11

ILM Facilities within the Database ..............................................................................................12

ILM Solution Overview ............................................................................... 12

ILM Planning Process Guidelines .............................................................. 14

Deciding Data Retention Periods................................................................................................14

Strategies for Data Retention Periods ........................................................................................15

3 W HITE PAPER / ILM Planning Guide

 Setting ILM_DT...........................................................................................................................15

Optimizing Eligibility ....................................................................................................................16

Scheduling the ILM Crawlers ......................................................................................................16

Deciding your storage options ....................................................................................................16

ILM Implementation Recommendations .................................................... 17

General Recommendations ........................................................................................................17

Implementing ILM on a database with history.............................................................................18

Conversion Considerations ........................................................................................................18

Using Enterprise Manager for ILM ..............................................................................................19

Example Scenarios .................................................................................... 19

Simple Archiving .........................................................................................................................19

Variation Of Simple Archiving .....................................................................................................20

Complex Archiving......................................................................................................................21

Frequently Asked Questions ...................................................................... 22

Conclusion ................................................................................................. 23

4 W HITE PAPER / ILM Planning Guide

INTRODUCTION

Information Lifecycle Management is a set of policies, tools and technologies

to allow implementations to optimize the storage and management of
substantial amounts transaction data to minimize costs and comply with
business rules.

WHAT IS INFORMATION LIFECYCLE MANAGEMENT?

Information Lifecycle Management is a set of techniques and technologies available from Oracle that
assist in managing the lifecycle of data to support business needs and minimize storage costs. The key
to Information Lifecycle Management is working with the business to ensure that data that the business
regards as active is available on appropriate hardware whilst data that is less active or dormant, in terms
of business activity, is managed in a effective way in terms of storage costs.

Data in the product is added and updated by the business on an ongoing basis. Over time, this data
grows and the business activity on individual data will vary with its timeliness, business activity and data
retention policies. As data becomes less active in terms of the business, it needs to be stored in a cost-
effective way to ensure cost minimization whilst allowing the business to access the data as needed for
analysis or auditing purposes.

Information Lifecycle Management respects the applications and business needs for accessing data
over its lifecycle while giving options to IT groups to minimize storage costs effectively over that life.

As the Oracle Database stores and manages data on an ongoing basis, therefore it is logical that
Information Lifecycle Management uses the inbuilt facilities and various database options to offer
effective storage management solutions to manage that data.

Note: This whitepaper is a planning guide and introduction to the capability. For detailed of information
for each product refer to the DBA Guides shipped with the product. This will outline the objects that are
covered by the ILM solution.

Why is Information Lifecycle Management Important?

Over time data is continually added to the database through transaction processing, which increases
the amount of storage over time. The amount of growth will depend on volume of transactions and the
size of the implementation. If no ILM solution is implemented, the growth of data storage would be
unchecked and at points in the lifecycle more storage would need to be added. This is costly over time
and if remained unchecked can lead to unexpected volumes and continual cost. In most situations, the
increase of storage costs is incurred when a storage capacity threshold is reached. The decision to
purchase new storage needs to be timed so that delivery timelines are considered without disruption to
operations. Without the effective management of older data, the storage costs only increase over time.
The figure below summarizes the situation when ILM is not implemented.

5 W HITE PAPER / ILM Planning Guide

 Storage Cost ($)

Storage Growth (GB/TB)

Time

Figure 1. Storage scenario without Information Lifecycle Management

The goal of effective storage management is to attempt to fix the amount of storage for the longest time
whilst keeping the data in the system needed for the business. This reduces the need for more storage
whilst supporting business operations. Using the various techniques and technologies available in ILM,
it is possible define the business expectations and map that to the storage capacity to keep within
constraints. This will attempt to keep within a set storage capacity. Ideally, data is removed from the
storage (or at least compressed) at a rate that covers the arrival of new data into storage. If there is a
shortfall in storage, as more data is added than removed or compressed, then additional storage may
be still be required but less frequently than there would of been in a non-ILM based solution. The figure
below illustrates the goal of an effective ILM solution.

Storage Cost ($)

Storage Growth (GB/TB)

Time

Figure 2. Storage scenario with Information Lifecycle Management

Deletion vs Information Lifecycle Management

One of the most frequent questions about using ILM is "why can't I just delete old data?". The answer is
to this question is complex. In most cases, data has a relatively simple lifecycle, so deletion seems to
be a viable alternative but not all data is the same. Complex business rules can sometimes determine
when a piece of data can be safely removed from the system, in other words, when the business has
finished with that data.

6 W HITE PAPER / ILM Planning Guide

Let's illustrate this with a real-life example. Bills and payments can be kept for a fixed amount of time, if
they balance correctly. It seems logical that perhaps you can simply remove bills and payments after a
fixed amount of time. For an example, let's assume that after 2 years you remove old bills and old
payment from the system. The DBA or implementer would write some process to remove data using
various means from the system and then you would think that would be fine. Now, for most bills and
payments that sounds perfectly logical. The issue is that in some cases you need to hold data for legal
reason past its use by date. For example, utilities must handle disputes correctly and if a dispute is still
active when the data ages, a deletion would in fact cause business issues if the the dispute is not closed.
This can cause integrity problems. In the ILM world, the outstanding business process would be
detected, and the data retention adjusted accordingly to maintain business operations.

The above scenario is just a single example but there are potentially other business processes that may
require individual records or objects to be managed differently. Having a fixed retention, without ILM, is
not flexible enough for a complete solution.

ILM CONCEPTS
Before outlining the planning guidelines, several key concepts in relation to Information Lifecycle
Management need to be understood.

Lifecycles
Data inside the product has a lifecycle, whether it is explicit or implied. As data is added and updated it
is considered active by the business. At some stage the data will be completed, with an explicit status
to indicate or an implied through association with other objects in the product. Once data is completed,
its usefulness to the business will slowly degrade as the record ages. In most cases, records that are
completed and retained are done so for legal or auditing purposes only. At this stage, the data needs to
be accessible to the business, even in read only mode, but needs efficient storage to save costs.

Eventually, the usefulness of the data will degrade to a point that the business does not need that data
anymore. At this stage, the data needs to be removed safely in a timely fashion.

Data therefore has a birth, period of business activity, lesser activity and finally dormancy which at which
time the data is removed from the system, as it is not needed by the business for any activity (legal or
otherwise).

The figure below illustrates and example of this lifecycle concept:

Time

Frequent Occasional
Active Access Access
Dormant

Figure 3. Example Lifecycle

Information Lifecycle Management respects this lifecycle and helps implement a complementary data
storage plan to minimize storage costs over this lifecycle.

7 W HITE PAPER / ILM Planning Guide

Data Types (Scope of ILM)
Information Lifecycle Management is not about managing every piece of data within a database. Data
growth is dependent on the type of data that is stored and managed as well as its business purpose.
Typically, Oracle Utilities Application Framework based products have several data types:

• Administration Data. This is data that defines the configuration and business rules for the
implementation. In terms of overall storage this is static and the data, whilst not updated often, is
accessed frequently throughout the life of the product. This data is broken up into several categories:

– Configuration Data. These are pieces of data that define the reference data used by the business.
Data such as values in dropdowns or business objects such as rates define the business rules in
data format, such as valid values. While this data can vary over time, its volume and its growth are
minimal in terms of total storage.

– Meta Data. This is data that drives the product at runtime such as menu items, display labels, table
structures. The volume and growth of this type of data are minimal in terms of total storage.

– ConfigTools Data. In Oracle Utilities Application Framework V2.x, the concept of configuration
data that contains programmatic like objects was introduced. This data defines the business rules
encapsulated in objects like Business Objects, Business Services, Scripts, and User Interface Maps
etc. The volume and growth of this type of minimal in terms of total storage.

• Master Data. This is data is that defines your main objects that drive the transactions in the system.
For example, Accounts in a billing system, meters in a metering system, crews in a Mobile Workforce
system, Assets in an Asset Management system etc. While this data can seem large in terms of
volume, its growth rate is typically under 10% and is deemed necessary for the business to operate
over time. In terms of storage, it is typically not managed using an Information Lifecycle Management
system as it defines the heart of the product.

Note: In Oracle Utilities Application Framework V4.3.0.6.0 and above, a new capability called Object
Erasure was introduced to handle the special processing required for managing Master Data. Refer
to the online documentation supplied with the relevant versions for more information.

• Transaction Data. Typically, transaction data is data that is related to the Master Data but records
the various business activities against that Master Data. For example, Payments, Bills, Meter Reads,
Activities, Tasks are transaction objects. These typically have high volume and high growth, though
the growth varies from object to object. This is the focus of the Information Lifecycle Management
based solution as they are the ideal objects to manage.

The figure below summarizes the data types and scope of the Information Lifecycle Management based
solution:

8 W HITE PAPER / ILM Planning Guide

 Configuration Data (e.g. rates)

Administration Data SDK Meta Data Low Volume,

Low Growth Not candidates
Data Types

for ILM
ConfigTools Data

Master Data Low Growth

High Volume, Candidate for

Transaction Data
High Growth ILM

Figure 4. Data Types and their relationship to ILM

Data Activity
One of the key concepts in Information Lifecycle Management is how active the data is within the
business. There are several definitions that need to be understood in terms of activity:

• In terms of the Oracle Database, activity is defined as INSERT, UPDATE or DELETE SQL statements
against individual rows. By default, access via SELECT statements are not considered activity on data
as Information Lifecycle Management supports read activities independent of the storage regime
used. In Oracle Database 12c and above, a new set of features such as Heat Maps and Automatic
Data Optimization (ADO) allows Database Administrators to track actual data activity in a business to
optimize the storage even further (including SELECT access, if necessary).

• For Information Lifecycle Management planning purposes, activity is defined as the period where the
business needs access to the data, regardless of its state, to support its business activities. For
example, in bills are considered active while the business needs the ability to re-bill (if necessary) or
to support estimation calculations. This will vary from object to object and implementation to
implementation.

• Activity in data becomes an important concept when planning the data retention policies that will drive
the business and technical sides of the Information Lifecycle Management solution.

Business Versus Technical Aspects

The Information Lifecycle Management based solution is a combination of a business solution and a
technical solution. These aspects work together to form a total data management solution that meets
the business requirements whilst ensuring storage of that data is cost effective and optimal.

The business aspects of the solution are defining the data retention policies for individual objects. The
business needs define how long individual objects are to be considered active within the business and
the business rules that govern them to ensure data consistency. Even though data has been in a
database a long time, it may be still active within the business. For example, a payment may be in
dispute for a long time and is still active and therefore should not be archived. The business will define
these aspects within the product itself to lock in the data retention policies.

The technical aspect mirrors the business aspect but consider the storage architecture to implement
cost savings. The technical solution takes the business requirements and takes them into consideration
when designing a compression and partitioning solution to optimize the storage. The technical aspects
can apply to data regardless of its state; in other words, business active data can be optimized as well
if required.

9 W HITE PAPER / ILM Planning Guide

 Business Rules, Legislation, Market Rules etc
Business

Transaction Data

Technology
Storage and Technology Solutions

Figure 5. Business and Technology solution in ILM

Simple versus Complex Solutions

Data retention and lifecycles supported by the Information Lifecycle Management based solution can be
as simple and as complex to meet business requirements but optimize storage costs.

Typically, solutions are wide ranging but there are several common extremes:

• Simple. After the business has deemed the data as dormant the data is removed using the Information
Lifecycle Management based solution. For example, implementations may wish several years online
and anything older than that time is removed from the database. This is a very simple design with
Information Lifecycle Management providing the business to define the cut off period and the technical
aspects moving that data to an archive partition for removal. The following figure illustrates an
example of this extreme:

Time

Active Dormant

Figure 6. Simple ILM scenario

• Complex. The business might want to define several stages for data to migrate through as a business
requirement. In this case, data would be defined as active, less active, occasional access and then
finally dormant. The number of stages will vary with the individual object and its relevance to the
business. At each stage it is possible to allocate lower cost hardware or different approaches to
minimize the storage costs but continuing to make the data available within the database. The
following figure illustrates an example of this extreme:

Time

Frequent Occasional
Active Access Access
Dormant

Figure 7. Complex ILM scenario

10 W HITE PAPER / ILM Planning Guide

No single approach is typically used at an implementation with various approaches, including numerous
variations on the above extremes, on individual objects, adopted.

DATA MANAGEMENT SOLUTION OVERVIEW

The Information Lifecycle Management based solution is based upon functionality in the Oracle Utilities
Application Framework and Information Lifecycle Management based technology built into the Oracle
Database. This provides a comprehensive solution for managing the lifecycle of data for the business
and providing options for minimizing data storage footprint.

Features and Functions

The Information Lifecycle Management based solution has several features and functions:

• The transaction candidates for the Information Lifecycle Management based solution are provided
with the product including all the related metadata, algorithms and initial solution scripts.

• Provides the ability for the business to define the data retention rules on an individual object basis.

• Provides the ability for the eligibility of individual instances of objects to be assessed for inclusion for
data management.

• Provides the ability to assess the business rules governing individual instances of object to ensure
their removal will not adversely impact the business operations and ensure data integrity.

• Works with various database technologies including compression (basic and advanced) and
partitioning to offer storage optimization options.

• Works with database features such as Automatic Storage Management (ASM) to interface to storage
solutions.

• Works with advanced data management capabilities available in Oracle Database 12c and above
including Heat Maps and Automatic Data Optimization.

ILM Facilities within the Product

The Oracle Utilities Application Framework has been enhanced to include additional facilities and
features to allow the implementation of the Information Lifecycle Management based solution as well as
communicate the business data retention periods. The Information Lifecycle Management facilities
within the Oracle Utilities Application Framework include:

• A Master Configuration object has been added to enable and control the Information Lifecycle
Management based solution. This allows a global data retention period to be implemented and identify
the transactional objects that have been enabled for data management activities.

• A set of ILM specific Maintenance Object Options have been added to allow individual object
configurations for the Information Lifecycle Management based solution.

• A set of ILM eligibility algorithms have been added to determine the eligibility of individual instances
of an object for data management.

• A set of background processes have been added to implement the eligibility algorithms. These are
called ILM Crawlers. A controlling background process is also provided called the ILM Crawler Initiator
which allows all the ILM Crawlers to be scheduled in one initiation.

11 W HITE PAPER / ILM Planning Guide

ILM Facilities within the Database
The Oracle Database contains several key technologies to manage the lifecycle of the data using
Information Lifecycle Management principles. The following facilities are available within the database:

• Compression. The Oracle Database introduced compression into Oracle 9i and above. This feature
allows storage to be reduced that is transparent to the applications. Data that is compressed can co-
exist with non-compressed data in the same block from disk to memory in buffer caches. There are
several compression options:

– Basic Compression. This compression is the compression shipped with the Oracle Database in
all editions. The basic compression facilities in the product are optimized for DSS applications only.

– Advanced Compression. This is an advanced option that provides higher level of compression as
well as additional compression algorithms that are optimized for specific operations.

– Exadata Hybrid Columnar Compression. Implementations using Oracle Exadata can utilize this
feature for advanced hardware accelerated higher compression levels that are optimized for the
hardware.

• Partitioning. Oracle supports separation of a table into segments which are allocated to different
storage devices to promote data management and increased processing parallelism.

• Automatic Storage Management (ASM). Oracle supports low level integration with hardware by
providing a Volume Manager and File System in the form of Automatic Storage Management (ASM).
This allows Database Administrators the ability to control, at a low level, the storage options for the
database. Use of this facility is optional for Information Lifecycle Management but can enhance the
level of control if used.

• Heat Map (Oracle Database 12c and above). In Oracle Database 12c, a new Heat Map feature was
introduced to allow Database Administrators to track the business activity on a table and row level to
ascertain actual activity versus planned activity. This tool will allow Database Administrators to identify
additional cost savings and further optimize the technical side of the solution. Use of this feature is
optional but encouraged to provide additional planning information for Database Administrators.

• Automatic Data Optimization (ADO) (Oracle Database 12c and above). In Oracle Database 12c,
a new facility to allow definitions for compression and other data management events on Data
Definition Language statements (such as CREATE TABLE) was introduced. This allows Database
Administrators to provide simple instructions for the database to manage data lifecycle. This
technology, while optional, allows additional flexibility in implementing Information Lifecycle
Management based approaches.

Note: For more information about the Information Lifecycle Management facilities refer to the
Information Lifecycle Management for Business Data whitepaper.

ILM SOLUTION OVERVIEW

The Information Lifecycle Management data management solution is a combination of changes to the
Oracle Utilities Application Framework and data management solutions within the Oracle Database. The
solution works in the following way:

• A Master Configuration record for ILM Configuration is provided to set the Default Retention Days.

12 W HITE PAPER / ILM Planning Guide

• Several Maintenance Object Options and ILM Eligibility Algorithms are configured on Maintenance
Object enabled for data management.

• The ILM_DT is set to the current date at record creation time and ILM_ARCH_SW set to N to initiate
the record.

Note: The ILM_DT may be altered by algorithms or batch process if reassessment is required or
implementation of complex data management rules.

A set of batch orientated ILM Crawlers and an ILM Crawler Initiator are provided for each Information
Lifecycle Management enabled object. The ILM Crawler is scheduled to execute and does the following:

• For all records where the current date is less than the ILM_DT + the Retention Period, where the
ILM_ARCH_SW is set to N the ILM Eligibility algorithms are executed to assess eligibility on each
instance of the object.

• If the ILM Eligibility algorithm assesses the object as eligible for data management, it will return true
and set the ILM_ARCH_SW to Y. Conversely of the ILM Eligibility algorithm deems the object not
eligible, returning false, then the current values are retained for future reassessment.

Once the ILM_ARCH_SW is set to Y, the ILM_DT has aged sufficiently or business usage on a row has
waned then the Information Lifecycle Management facilities, via Enterprise Manager, within the
database can be configured to implement any of the follow:

• Implement compression based upon data usage. This can be basic compression, advanced
compression or Hybrid Columnar Compression (for Oracle ExaData implementations).

• Implement partitioning based upon data groups. The number of partitions can vary due to the
requirements and storage architecture to separate active data from less active data.

• Allocate partitions to relevant storage to implement storage cost savings.

• Implement transportable tablespaces to implement effective data removal.

• Use features in Oracle Database 12c and above, such as Heat Maps and Automatic Data Optimization
to further optimize storage.

The following figure summarizes the processing flow:

13 W HITE PAPER / ILM Planning Guide

 Master Configuration Record Default Retention Period

Maintenance Object ILM Retention Period In Days

ILM Crawler Batch Control
ILM Eligibility Algorithm
etc

ILM Crawler
Time

Frequent Occasional
Active Access Access
Dormant

Active Dormant

Retention
ILM_DT set
Period
ILM_DT +
Compression, Partitioning, ADO
ILM_ARCH_SW set
Retention
Period
reached

Figure 8. Summary of the ILM Solution

ILM PLANNING PROCESS GUIDELINES

Note: To implement the Information Lifecycle Management based approach it is recommended to follow
the Information Lifecycle Management installation and solution overview documentation provided with
your product release.

Note: For customers using Oracle Database 12c Enterprise Edition and above, it is recommended to
use the Database Instance target type within Oracle Enterprise Manager to perform Information Lifecycle
Management activities.

Note: In previous releases of the Oracle Database, an addon ILM Assistant was provided. This is not
required as similar functionality is available via the database features of Oracle Enterprise Manager.

To plan your Information Lifecycle Management based data management solution there are several key
steps and design concepts that must be taken in account.

Deciding Data Retention Periods

The first step to designing the solution is for the business to decide and configure the data retention
periods for the business, inside the Oracle Utilities Application Framework based product. A Default
Retention Days, within the ILM Configuration Master Configuration record, needs to be set and if
required, data retention periods for individual objects that are data managed by the Information Lifecycle
Management based solution.

When deciding data retention periods, the following considerations should be considered:

• Consider the duration business activity against the individual object as an indicator for potential data
retention policies. Business activity is regular creation or update of the object during business
processing. For example, rebills or recalculation timeframes may set this value.

14 W HITE PAPER / ILM Planning Guide

• Consider legal requirements, such as the Sarbanes–Oxley Act and/or Payment Card Industry Data
Security, for the data to set data retention periods. In some implementations there are regulations that
govern the lifecycle of data in systems. The business interpretation of these regulations also needs to
be considered. For example, tax regulations in some countries sets the data retention rates but this
does not necessarily mean that it must be available online for the total duration; it may be satisfied
partially online and partially in other compatible formats.

• Consider potential customer dispute processes in your data retention policy. Whilst the business may
be finished with some data, if there may be a dispute timeframe that means that the data must be
retained for addition times.

• Once the data retention period has been agreed the Maintenance Object Options need to be
configured.

Strategies for Data Retention Periods

There are several strategies when setting the Data Retention Period on objects:

• Simplistic Solutions. The business can set the total data retention period for the data within the
product and the technical phase of the Information Lifecycle Management based solution can simply
archive the records after the ILM Crawler has set the ILM_ARCH_SW to Y. This is the simplest setup
using Oracle Enterprise Manager to implement a storage configuration to match the total retention
period and the archive/purge action to remove the data after this period. In this case the data retention
period would encapsulate any legal and business requirements.

• More complex solutions. The data retention period, set globally or on individual objects, is flexible
in a more complex solution giving the implementation more options for data retention and flexible
storage options. There are several strategies applicable to this approach:

– Setting the Data Retention Days low. This strategy allows for the business to define the absolute
minimum period they need the data active. The technical phase of the solution is then able to
provide additional storage options to allow the business more time to access the data, for example
for compliance, whilst minimizing storage at an earlier time in the lifecycle. The technical phase of
the solution can then offer multiple storage options over the lifecycle before the data is eventually
deemed dormant.

– Setting the Data Retention Days high. This strategy utilizes the same principles as the simplistic
solution, but the technical phase of the solution can implement multiple storage strategies within
the data retention period to allow for storage optimization at an earlier stage than the simplistic
solution.

Setting ILM_DT
Note: For new records ILM_DT is automatically defaulted to the row creation date within the database.

They key for the Information Lifecycle Management based solution is setting the ILM_DT column. By
default, the ILM_DT is set at record creation time. This column controls the date that ILM Crawler will
start to evaluate the record for setting the ILM_ARCH_SW, appropriately, to determine when the handover
between the business and technical aspects is indicated.

The ILM_DT can be altered by any algorithm, business process etc inside the product to implement
flexible data retention policies. For example, the date may be extended if a dispute process is still in
progress after the original ILM_DT has been set.

15 W HITE PAPER / ILM Planning Guide

Optimizing Eligibility
By default, the Information Lifecycle Management based solution includes an ILM Eligibility algorithm
that allows the ILM Crawler associated on the Maintenance Object to evaluate the object to set the
ILM_ARCH_SW after the ILM_DT. Each object ships with a specific algorithm optimized for that object or
uses the generic ILM Eligibility algorithm (F1-ILMELIG).

When the difference between ILM_DT and the current date is greater than retention period configured,
then the record is eligible to be assessed by the ILM Crawler. Effectively, the age of the record has
dictated that the business has finished with the data and it must be assessed to ascertain if there are
any outstanding related objects.

The algorithm provides the following processing:

• Using meta data any related objects (parent and dependent) are checked to see if they are also in a
final status. If any related object is in a non-complete state (such as an initial or interim state) then the
ILM_ARCH_SW cannot be set to Y as there is still outstanding work. The ILM Crawler will continually
check this with each subsequent execution.

• For some objects, specialized checks and balances are processed before the ILM_ARCH_SW is set
to Y. For example, financial records may need to be balanced before archival is possible. The details
of the specific checks implemented by the algorithm are summarized on the Algorithm metadata in
the Algorithm Type Description.

These shipped algorithms can be altered to add additional eligibility criteria to determine under what
conditions the ILM_ARCH_SW is set to Y.

Note: The comments on the algorithm meta-data describe the conditions checked for the object by that
algorithm. Refer to those comments for details.

Scheduling the ILM Crawlers

The ILM Crawler is a special type of batch process that is executed per object to assess the eligibility of
objects by using the ILM_DT and setting the ILM_ARCH_SW. These batch processes need to be
scheduled using the following recommendations:

• The ILM Crawlers can be scheduled as a group using the ILM Crawler Initiator (F1-ILMIN) or
individually using the individual Batch Controls associated with the individual objects .

• The ILM Crawlers are recommended to be executed daily to reassess objects that may become
eligible. They can be scheduled to start at the business day or the end.

• The ILM Crawlers are different to other batch processes. Typically, batch processes start and end
after a duration. The ILM Crawler is started and will wait in the background to assess data on a
continuous basis. It will only end if has been stopped manually using jmxbatchclient or otherwise.
Give their unique execution method, Oracle recommends ILM Crawlers should be executed in their
own threadpool.

• The cutoffDate parameter can be used to run the ILM Crawler for segments in the past. This
provides a mechanism for aging historical data over time. Refer to the Implementing ILM on a
database with history for guidelines in this area.

Deciding your storage options

16 W HITE PAPER / ILM Planning Guide

Once the product configuration has been configured the storage options can be configured using Oracle
Enterprise Manager (and/or commands). There are several guidelines to consider when deciding your
storage options:

• The number of partitions to implement, at a minimum, should match the stages of data activity your
business wants to implement. Implementations using Real Application Clustering should consider
making the number of partitions also be a multiple of the number of nodes in the cluster.

• Each partition should be allocated to appropriate storage either using your SAN software, Oracle
Automatic Storage Management or manually. This storage can be of varying costs or not.

• Create a Lifecycle that matches your business requirements. The last stage of the lifecycle should be
purging the data (to remove it from the system).

• Consider compression based upon activity of the data. Whilst basic compression can save space,
advanced compression and Hybrid Columnar Compression can offer greater flexibility as well as
greater levels of compression.

• Compression can be considered for data where the ILM_DT is less than the current date where usage
has been reduced to implement additional savings. If this option is chosen and the data that is
compressed, then performance degradation may occur if data updates are performed on that
compressed data.

• Dormant data should be allocated to Archive.

ILM IMPLEMENTATION RECOMMENDATIONS

When implementing the Information Lifecycle Management there are several recommendations to
optimize the setup.

General Recommendations
These are general recommendations when implementing the Information Lifecycle Management based
solution:

• Assess the objects that your implementation need to be managed using Information Lifecycle
Management. Not all the objects supplied are applicable for all implementations.

• If possible, only use data retention periods on individual objects if they differ from the Default Retention
Days. A value of zero (0) on the Maintenance Object Option Retention Days or not specifying that
option at all, will default to the Default Retention Days.

• Remember data storage costs are not restricted to the amount of data stored but also the cost of
backing up that data and the power costs for running the devices storing that data. Lower cost disk
can use less power which can represent further savings that just from raw storage cost.

• Each table to be managed using Information Lifecycle Management should be housed in its own
tablespace. This will prevent other objects being affected that may or may not be Information Lifecycle
Management enabled or share the same data storage characteristics.

• Only remove data where the ILM_ARCH_SW is set to Y. This flag is set if the object is consider not
needed by the business and is safe to remove, from a business point of view. Failure to do so may
result in unexpected behavior from the product.

17 W HITE PAPER / ILM Planning Guide

• Compression is a simpler data management strategy than Partitioning. Generally, compress before
you partition. Using Oracle Automatic Data Optimization in Oracle Database 12c or above, improves
automation of compression,

• If Advanced Compression is used, then OLTP compression is recommended.

• If read only tablespaces are being considered for the solution, then ensure that the data relegated to
that tablespace is not ever going to be updated by the business. If the business attempts to update
the record for any reason it will generate a database error. Also remember that if the ILM_ARCH_SW
is set to N and the data is in a read only tablespace the ILM Crawler will fail whilst attempting to update
the flag.

• Transportable tablespaces have an advantage over standard partitions. They can be restored later, if
needed and they act like a garbage bin where data is deposited and can be emptied at a later stage.
This last feature allows the data to be retained longer, just in case, but also minimizes manual effort
as the disconnect event can be performed over longer periods (say once a month or once a quarter).

• If you already have implemented partitioning for the product, then use the ILM partitioning
recommended as the main partitioning and then implement sub partitioning for your existing
partitioning strategy. This will maximize the benefits of both approaches.

Implementing ILM on a database with history

For implementation with substantial amounts of data already in the database there are a few techniques
for implementing the Information Lifecycle Management based solution:

• It is recommended to implement Information Lifecycle Management in stages for each object to

minimize the impact on the business of the processing.

• Divide your existing data by setting the ILM_DT on the data in partitions.

• Use cutoffDate values on individual ILM Crawler batch processes to process each partition to set
the group of data to process. This value overrides the date used by the individual crawler for the
assessment of data (by default it uses the system date).

Note: The ILM Crawler Initiator does not support cutoffDate values. It is recommended to submit
individual ILM Crawlers to implement this technique.

After processing each group of data, it is recommended to to assess the ILM setup to reflect the data
changes.

Conversion Considerations
If your implementation is converting from a legacy system, then the ILM_DT and ILM_ARCH_SW should
be set during that conversion process for historical data using the following guidelines:

• ILM_ARCH_SW should be set to the default value of N. It makes no sense in converting data into an
archive state at initial loading. The ILM Crawler will re-evaluate the record at the appropriate juncture
and set the flag accordingly.

• The ILM_DT should be set appropriately to age the records according to your data retention needs.
Typically, ILM_DT is the defaulted to the record creation date but that may not be available in the
source system to set the ILM_DT appropriately. If this is the case, then set the ILM_DT to an
appropriate date (factor in when you want the data to be considered dormant).

18 W HITE PAPER / ILM Planning Guide

Using Enterprise Manager for ILM
In past releases of the Oracle Database, an optional component called Database Control was shipped
to allow database administrators to manage the database using a browser interface. In Oracle Database
12c, Database Control has been replaced with the base functionality of Oracle Enterprise Manager.

After Oracle Enterprise Manager is installed and the database instance is registered as a target, the
database can be managed using Oracle Enterprise Manager. All the administration functions that a
database administrator can be performed from this console.

Note: Use of Oracle Enterprise Manager, whilst recommended by Oracle, is optional. DBA level
commands or alternatives may be used with varying results.

EXAMPLE SCENARIOS
There are many ways Information Lifecycle Management can be implemented to help satisfy your
business data management requirements. There are several common scenarios that are discussed in
this section to help plan the data management solution to suit your needs.

Note: The scenarios listed in this section are not exhaustible but represent common data management
scenarios.

Simple Archiving
The simplest scenario is to implement two data groups, one for active data with no compression and
one for archived/purged data. Essentially the business defines a simple configuration where the data is
either required or can be deleted. This scenario is useful for several business needs:

• The data storage costs are only minimized through the deletion of data when it is dormant.

• The cost of storage is not a concern even when business activity on data may vary over time.

• The business is not sure of their data retention needs and wants a very simple data management
solution.

• This scenario assumes the minimum setup for partitioning with one partition for active data and a
second for purging data, which can either be a transportable tablespace or similar within the
partitioning defined in Oracle Enterprise Manger

• Compression is not featured in this scenario to minimize effort. If compression is to be used, then
refer to the Variation Of Simple Archiving for implementation details.

This scenario is summarized in the following figure:

19 W HITE PAPER / ILM Planning Guide

 Record No Longer
Needed by
Record Created Business Record Removed

Tablespace Transportable Tablespace

No Compression Purge/Disconnect

Active Dormant

ILM_DT set ILM_ARCH_SW set

Figure 9. Simple Scenario

To implement this scenario the following configuration should be implemented:

• Set the Data Retention Days globally or on individual objects using the Master Configuration and
Maintenance Object options respectively. This should be set to the value that represents when the
business does not require the data anymore and it can be archived if the ILM Eligibility indicates it
can be archived.

• Within the Oracle enterprise Manager (or commands) create two partitions/data groups based upon
the ILM_DT with the data retention period considered as per the product DBA Guide. The last partition
should either be transportable or be set to an appropriate action.

Note: Remember only purge data where the ILM_ARCH_SW is set to Y and has been aged accordingly.
Failure to do so may result in unexpected behavior from the product.

If the tablespace is transportable then disconnect it at the appropriate time.

Variation Of Simple Archiving

Whilst the simple scenario is simple it does realize storage costs savings till after the data has been
removed. In most cases, if the new data is added at the same volume or at a greater volume than those
being removed then storage costs will in fact increase, albeit rather slowly.

One variation to address this is to attempt to introduce storage minimization techniques earlier in the
data lifecycle. One such technique is to introduce compression (base compression, advanced
compression or Hybrid Columnar compression) when the data is considered read only for the business.

Note: It is recommended not to compress data that is actively being updated as that may impact
performance. Read only data can be safely compressed as it does not affect performance, in fact it
increases performance.

Note: Setup of this scenario is much easier using Oracle Database 12c or above, using the Automatic
Data Optimization feature

This scenario is very similar to the Simple Archiving scenario with the following differences:

20 W HITE PAPER / ILM Planning Guide

• If the business can determine when the data is no longer updated by the end users, then a tablespace
definition can be constructed within Oracle Enterprise Manager to separate that data into that
tablespace. This does not have to affect the Data Retention Days at all.

• The tablespace can be allocated to lower cost storage or be compressed at a tablespace level.

• Customers using Oracle 12c can use Automatic Data Optimization to compress the data regardless
of the tablespace allocations to provide higher levels of data storage savings.

This scenario is summarized in the figure below:

Record No Longer Record No Longer

Updated by Needed by
Record Created Business Business Record Removed

Tablespace Transportable Tablespace

No Compression Compression Purge/Disconnect

Active Dormant

ILM_DT set ILM_ARCH_SW set

Figure 10. Variation on simple scenario

To implement this scenario the following additional configuration should be implemented:

• Within Oracle Enterprise Manager create a partition for the data that is not to be updated. Allocate a
separate tablespace for that partition.

• Allocate that data group to a lower cost storage and/or compression.

• In Oracle Database 12c or above, configure Automatic Data Optimization with the relevant
compression rules.

All the implementation of the advice from the Simple Archiving scenario should be reused.

Complex Archiving
Note: This scenario has many variations, this is just presented as an example.

One of the last common scenarios is where the data retention plans are complex and require a set of
data groups to implement data retention plans. The focus of this scenario is to provide the business
access to the data if possible but still take advantage of the storage cost saving capabilities.

The key differentiators for this scenario are:

21 W HITE PAPER / ILM Planning Guide

• It is ideal where the business wants varied access to data over time or there are legal requirements
that require longer retention periods than the business requires.

• This scenario offers the maximum option for implementing flexibility through combinations of
compression, storage options and tablespace definitions. This configuration takes maximum
advantage where implementations have access to several types of storage with differing costs.

• This scenario allows storage of data in cost effective storage based on the age and activity on data at
decreasing costing levels.

• This scenario is ideal for customers where parts of the business have different data retention needs
which cannot be resolved by a simple solution but needs to reduce data storage costs.

This scenario is summarized in the figure below:

Record No Longer Record No Longer

Updated by Needed by
Record Created Business Business Record Removed

Tablespace Read Only Tablespace Read Only Tablespace Transportable Tablespace

No Compression Compression Compression Purge/Disconnect

Less Occasional
Active Access Dormant
Active

ILM_Date set ILM_ARCH_SW set

Figure 11. Data Types and their relationship to ILM

To implement this scenario the following configuration settings should be implemented:

• The number of partitions will vary according to the storage facilities, cost minimization strategy and
business/legal requirements. For example, if Advanced Compression is used then varied amounts of
compression, associated with each data group, can be configured as the data ages.

• For each partition decide the level compression, data access rights and storage attributes appropriate
for your storage solution and the desired storage savings.

• Advanced Compression and Hybrid Columnar Compression offers greater levels of compression
options and flexible configurations.

FREQUENTLY ASKED QUESTIONS

There are frequent questions around for implementing ILM. This is a summary of the most frequent
questions and their associated answers.

Frequently Asked Questions

STEP PROCESS

22 W HITE PAPER / ILM Planning Guide

 What do I need to license on
the database for this solution?
Can I add my own objects to
the solution?
What if I have partitioned the
same tables already?
When is the best time for
implementing ILM?
Can I use the In-Database
Archiving feature of Oracle 12c
Database and above with this
facility?
How do I deal with the ILM
Crawlers?
I am an existing customer with
lots of data already, how can I
implement this facility?
I noticed that ILM_DT is only on
certain tables, why is that?
What training is available for
the database side of the
solution?
CONCLUSION
The ILM solution for Oracle Utilities Application Framework products marries the business requirements
with the traditional storage based solutions to provide a comprehensive storage based solution that
minimizes storage costs and ensures data is accessible to the business users.
23 W HITE PAPER / ILM Planning Guide
You must license the following on the database:
• Partitioning. This is required for archiving data and providing cost
reduction for storage.
You can optionally license the following:
• Advanced Compression. Offers flexible and greater levels of
compression.
No. At the present release custom objects or additional objects are not
supported by the solution.
This is not an issue. The prime partitioning method should use the new
ILM columns while existing partitioning becomes sub partitions. This will
allow the ILM facilities to manage the lifecycle but also allow you to
retain the benefits of your existing partitioning.
Typically, data management cost issues occur sometime after initial
implementation, but this can be implemented at the initial
implementation time as data is added continuously after initial
implementation. Data storage costs are typically a driver for
implementation of this solution and whenever storage costs are too
costly for the business signifies a trigger to start ILM activities.
Yes, it can be used in partnership with this facility. For example, you
can use the feature to hide records prior to removing them. They can
act as insurance with the business before you remove them
permanently. Use of this facility will only hide the data from the product
not save storage.
Typically, the ILM Crawlers are scheduled like any batch process in the
product. They can run on scheduled basis or on continuous basis.
Given they run continuous they can be manually started and only
stopped for maintenance activities such as backups etc.
You can use the ILM Crawler Initiator to kick off all the ILM Crawlers
together.
Yes, refer to Implementing ILM on a database with history for more
details.
The ILM solution is driven by the parent tables within the objects in the
product. The fate of the children tables in an object is dictated by the
fate of the parent table. This is intentional behavior and optimizes the
storage requirements for ILM.
The DBA courses for each version include ILM specific materials
including Heat Maps, Partitioning, Automatic Data Optimization. Refer
to the local Oracle University site for classroom and online training
available.
24 W HITE PAPER / ILM Planning Guide
ORACLE CORPORATION

Worldwide Headquarters
500 Oracle Parkway, Redwood Shores, CA 94065 USA

Worldwide Inquiries
TELE + 1.650.506.7000 + 1.800.ORACLE1
FAX + 1.650.506.7200
oracle.com

CONNECT W ITH US
Call +1.800.ORACLE1 or visit oracle.com. Outside North America, find your local office at oracle.com/contact.

blogs.oracle.com/theshortenspot www.linkedin.com/in/theshortenspot twitter.com/theshortenspot

Copyright © 2019, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are
subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed
orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be
reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or
registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of
Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0119
ILM Planning Guide - Oracle Utilities Application Framework
January 2019