64-bit Aggregates in Data ONTAP

Overview and Best Practices

GV Govindasamy
Technical Marketing Engineer
February 2015

1 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Agenda

1) Aggregate Evolution
2) Overview
3) In-place expansion to 64-bit Aggregates
4) Best Practices
5) Architecture Deep Dive
6) Interoperability
7) Performance during in-place expansion
8) References

2 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
Aggregate Evolution

3 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
 NetApp aggregate evolution
Timeline Evolution
7.0 The introduction of aggregates. A container that can hold one or more NetApp
2005 FlexVol volumes. The initial format is 32-bit and limited to maximum size of 16TB.

The introduction of support for 64-bit aggregates which can support larger FlexVol
8.0 7-Mode
volumes. Both 32-bit and 64-bit aggregates can coexist in the system. The new
8.0.1 Cluster
maximum size for aggregates and FlexVol volumes can be found at NetApp
2010-2011
Hardware Universe at https://hwu.netapp.com.
8.1
The default aggregate type is changed from 32-bit to 64-bit for new aggregates.
2012
8.2
In-place expansion of 32-bit aggregates to 64-bit introduced.
2013
8.1.4P4 / 8.2.1 In-place expansion procedure for 32-bit aggregates to 64-bit without requiring
2014 additional disks.
8.3
Deprecation of support for 32-bit aggregates. Only 64-bit aggregates are supported.
2015
4 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
Overview

5 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Overview of aggregates
Introduced in Data ONTAP 7.0

Aggregates provide flexibility and ease of use. Flexible Volumes

Aggregates enable abstract volume
management from disk management.
The 64-bit aggregates are introduced starting
with Data ONTAP 8.0 operating in 7-Mode Aggregate

and clustered Data ONTAP 8.0.1.

All Data ONTAP versions are supported.
No license is required.
RAID Groups
Starting in clustered Data ONTAP 8.3, the
support for 32-bit aggregates is deprecated.

6 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Benefits of 64-bit Aggregates
Key Advantages
Larger volume sizes
Some applications such as genomic research, seismic interpretation, satellite imagery, and PACS require
very large volumes.
NetApp FlexVol volumes that have the space guarantee set to Volume can be up to 90% of maximum
aggregate size
Ease of management
Large number of FlexVol can be hosted on fewer aggregates, which simplifies the management.
Data consolidation is possible.

High performance with larger SATA drives

More data drives per aggregate can boost application performance.

Greater storage efficiency with larger SATA drives

14+2 NetApp RAID DP group with drives of 1TB and larger is supported.

7 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Benefits of 64-bit Aggregates (continued)
Key advantages

Better scalability for future growth

Bigger aggregates and volumes offer a better way to plan and manage data growth.
64-bit format can scale to millions of terabytes.
Power and cost savings
Using larger drives typically means using fewer drives.
Using fewer drives means less power, cooling, and physical space consumption.

8 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Maximum aggregate size
Aggregate size threshold

32-bit Aggregates
Aggregate uses 32-bit block pointers and is limited by its addressable space of 16TB.
64-bit Aggregates
Aggregate uses 64-bit block pointers and addressable space to address millions of terabytes.
The maximum size threshold depends on the hardware platform.
System capacity
Memory
Hardware configuration
The actual supported size for each platform can be found in NetApp Hardware Universe at
http://hwu.netapp.com.

9 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Clustered Data ONTAP 8.3
Upgrade path

Upgrade your existing cluster to 8.2.1 prior to upgrade to 8.3.

Expand all 32-bit aggregates to 64-bit aggregates.
Ensure that 32-bit NetApp Snapshot copies are either deleted and/or cycled off the
system.
Format NetApp SnapMirror relationships in 64-bit by expanding the residing aggregates
of the source and destination volumes.
NetApp recommends that source aggregates be expanded before destination aggregates.
Finally, upgrade the system to clustered Data ONTAP 8.3.
Starting in clustered Data ONTAP 8.3, only 64-bit aggregates are supported.
SnapMirror relationships also are supported in 64-bit formats.

10 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
In-place expansion to 64-bit aggregates

11 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
In-place expansion
Overview
Both 7-Mode (8.0) and clustered Data ONTAP (8.0.1) are supported.
No additional license is required.
In earlier releases of Data ONTAP (before 8.2.1), in-place expansion is triggered when
additional disks are added to the aggregate with the resulting new size >16TB.
Starting with Data ONTAP 8.2.1 or 8.1.4P4, in-place expansion can be triggered by an
administrator without requiring additional disks.
In-place expansion process is manually triggered by administrators.
In-place expansion does not happen automatically during ONTAP release upgrades.
Expansion process maintains checkpoints during the operation. If it is interrupted, the
process resumes after the cause for the interruption is fixed.

12 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
In-place expansion
Nondisruptive expansion
In-place expansion expands only metadata blocks and does not modify data blocks.
No data copy or migration is required.
Data access is maintained during the expansion.
This is nondisruptive to the clients. The aggregate configuration details such as the
storage efficiency features and the data protection relationships are not affected.
There is minimal impact on system throughput.
The Expansion process can take up to 2% additional space for the expansion of metadata
blocks
The actual size varies depending on the metadata size of the volume/aggregate. Typically, this is very
insignificant for most of the datasets.

13 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
In-place expansion
Requirements

Verify that free space is available to complete the expansion

Upgrading 32-bit blocks to 64-bit consumes more metadata in a volume or an
aggregate
The extra metadata required is minimal in most cases
Verify that 32-bit NetApp Snapshot copies do not exist on the
aggregate.
Expansion can be triggered only after the completion of the
prequalification scanner
This is a background scanner that calculates the number of additional blocks
needed for space-reserved files
The scanner is triggered automatically when you upgrade to NetApp Data
ONTAP 8.1 or newer
14 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
In-place expansion
Workflow

Add disks to grow the aggregate beyond 16TB Complete the command-line commands to
begin the expansion process

Background expansion starts on all volumes

in the aggregate (not SnapMirror
Monitor expansion progress (optional)
destinations)
New writes will be in 64-bit format

Expansion finishes on all volumes in the Expansion process finishes on all volumes
aggregate within the aggregate

Data ONTAP 8.1 and 8.2 Data ONTAP 8.2.1 (or 8.1.4P4)

15 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Caveats
In-place expansion limitations

NetApp FlexCache volumes must be destroyed before the expansion

process can start.
FlexCache volumes can be recreated after the expansion process has started
64-bit expansion is a one-way process.
32-bit NetApp Snapshot copies are not converted to 64-bit.
The administrator must delete them manually or wait for them to cycle off and
be replaced by a 64-bit Snapshot copy.
NetApp DataMotion for Volumes in 7-Mode across aggregate types is
not supported

16 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
32-bit and 64-bit aggregates support
Compatibility

8.0 8.0.1 8.2.1+

Data ONTAP 8.1 8.2 8.3
7-Mode Cluster 8.1.4P4
Default for new aggregates 32-bit 32-bit 64-bit 64-bit 64-bit 64-bit
32-bit support yes yes yes yes yes no
64-bit support yes yes yes yes yes yes
Coexistence of 32-bit and
yes yes yes yes yes no
64-bit aggregates
In-place expansion support no no yes yes yes n/a
In-place expansion without
no no no no yes n/a
requiring additional disks
64-bit root volume support no yes yes yes yes yes
17 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
Best practices

18 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Maximum sizes
Best practices: 64-Bit aggregate, RAID, size

Use 64-bit aggregates. Use in-place expansion or other data migration

methods to format data from existing 32-bit to 64-bit.
Use the default raid type (NetApp RAID DP) and refer to TR-3437,
Storage Subsystem Resiliency Guide, for recommended RAID group
sizes.
Refer to NetApp Hardware Universe at http://hwu.netapp.com for
maximum aggregate, file, LUN, and root volume size
recommendations.

19 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Consider FlashCache for high random workloads
Best practices

High random workloads can lead to high memory usage when data set
involves metadata in 64-bit aggregates.
Monitor performance during high activity.
Deploy appropriately sized NetApp FlashCache to improve read
performance following TR-3832, Flash Cache Best Practices Guide.

20 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Expansion process
Best practices

Ensure adequate space in aggregate and volumes before the

expansion process is started
NetApp recommends enabling compression only after expansion
process is complete.
Do not change volume autosize settings during the expansion process.
Destroy the NetApp FlexCache volumes in the aggregate to be
expanded.
Expand source aggregate before expanding destination aggregate so
that NetApp SnapMirror relationships perform optimally.

21 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
Architecture deep dive

22 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
32-bit aggregates
Using 32-bit pointers and addressing

Data ONTAP addresses the

disk in 4kB blocks
Each 4kB block address is 32
bits and is stored as such on
disk
Maximum possible aggregate
size is 232 x 4kB = 16TB

23 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
64-bit aggregates
Using 64-bit pointers and addressing

4KB block address has

increased to 64 bits
Addressing format change is an
on-disk data structure change
Maximum possible size is >1
million petabytes
Aggregates thresholds are large
enough for the foreseeable
future without requiring on-disk
format or memory addressing
format changes

24 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
In-place expansion
Closer look

Its a process that converts 32-bit block pointers to 64-bit block pointers by re-
writing indirect (also known as metadata blocks) blocks in the aggregates and the
volumes
In-place expansion uses one of the two methods, depending on what occurs first:
A background process that converts the indirect blocks to use 64-bit block pointers. This happens
in the background and uses system resources without affecting clients. This starts after an
administrator triggered the expansion process.
Inline: When a client access their data on a volume/aggregate after expansion is triggered but
the accessed data has not been expanded yet, the involved metadata gets expanded to 64-bit
inline. This could result in minimum delay to clients.
Expansion process maintains checkpoints during the operation. The process
resumes from the last checkpoint at which it was interrupted after the issue is
fixed.
25
For example, interruptions might be caused by SFO, CFO and out-of-space conditions.
2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
32-bit to 64-bit expansion
Sample diagram

26 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
Interoperability

27 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
SnapMirror compatibility for aggregate formats
Clustered Data ONTAP matrix
Destination Aggregate
Applies to Source
Volume 8.0.x 8.0.x 8.1.x 8.1.x 8.2.x 8.2.x 8.3
asynchronous volume (32-Bit) (64-Bit) (32-Bit) (64-Bit) (32-Bit) (64-Bit) (64-bit)

SnapMirror, NetApp 8.0.x (32-bit) Yes No Yes No Yes No No

DataMotion for
8.0.x (64-bit) No Yes No Yes No Yes Yes
Volumes, and load-
sharing mirrors 8.1.x (32-bit) No No Yes Yes Yes Yes No

8.1.x (64-bit) No No Yes Yes Yes Yes Yes

8.2.x (32-bit) No No No No Yes Yes No

8.2.x (64-bit) No No No No Yes Yes Yes

8.3 (64-bit) No No No No No No Yes

28 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
 SnapMirror compatibility for aggregate formats
Data ONTAP operating in 7-Mode
Destination Aggregate
Source
Applies to Volume 7.3.x 8.0.x 8.0.x 8.1.x 8.1.x 8.2.x 8.2.x
(32-Bit) (32-Bit) (64-Bit) (32-Bit) (64-Bit) (32-Bit) (64-Bit)
NetApp asynchronous
volume SnapMirror 7.3.x (32-bit) Yes1 Yes1 No Yes1 Yes1 Yes1 Yes1
Synchronous Volume
SnapMirror 8.0.x (32-bit) No Yes No Yes1 Yes1 Yes1 Yes1
Vol Copy 8.0.x (64-bit) No No Yes Yes1 Yes1 Yes1 Yes1
Aggr Copy
8.1.x (32-bit) No No No Yes Yes1 Yes1 Yes1

8.1.x (64-bit) No No No Yes1 Yes Yes1 Yes

8.2.x (32-bit) No No No No No Yes Yes1

8.2.x (64-bit) No No No No No Yes1 Yes

1
Does not apply to NetApp DataMotion for Volumes, which was introduced in NetApp Data ONTAP 8.0.x and works only between aggregates of the same format.

29 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Volume SnapMirror: Use Case 1
64-bit aggregate as source and 32-bit aggregate as destination

Aggr Aggr

64-bit 32-bit

Vol Vol

64-bit 64-bit

A NetApp SnapMirror relationship from a 64-bit volume to a volume in a 32-bit

aggregate results in a 64-bit destination volume residing in a 32-bit aggregate.

30 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Volume SnapMirror: Use Case 2
32-bit aggregate as source and 64-bit aggregate as destination

Aggr Aggr

32-bit 64-bit

Vol Vol

32-bit 32-bit

A NetApp SnapMirror relationship from a 32-bit volume to a volume in a 64-bit

aggregate results in a 32-bit destination volume residing in a 64-bit aggregate
The SnapMirror destination is expanded to 64-bit on a SnapMirror break or if the source gets
expanded to 64-bit.
31 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Qtree SnapMirror, SnapVault, NDMP Copy (7-Mode)
Interoperability across different aggregate formats

Logical replication engine replicates data

independent of aggregate formats.
The features are:
Qtree SnapMirror QSM
NetApp SnapVault Destination
NDMPcopy Source 32-bit 64-bit
aggregate aggregate

32-bit aggregate YES YES

64-bit aggregate YES YES

32 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
SnapVault on clustered Data ONTAP
Interoperability across different aggregate formats

NetApp SnapVault software on clustered Data ONTAP works only

on 64-bit aggregates.

Destination
Source 32-bit 64-bit
aggregate aggregate

32-bit aggregate n/a n/a

64-bit aggregate n/a YES

33 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
Performance during in-place expansion

34 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Performance during In-place expansion
In-place expansion process impact on performance

There is minimal impact on system latency and throughput due to the overhead
of the inline 64-bit expansion process
The 64-bit expansion may initially affect performance for a shorter period. The
performance improves gradually as the expansion process progresses.
Initial performance hit of the 64-bit expansion process is observed for a short window (while the 64-bit
expansion process is estimating space requirements)
The following two slides show the impact of the 64-bit expansion on the SFS and
SPC-1 benchmarks
Performance tests measured system latency while throughput was maintained the same as baseline test
with 32-bit aggregates.

35 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
SFS benchmark: average latency and throughput
During in-place expansion

Initial short spike in latency

Expansion process starts

Baseline:total_ops
Baseline:sys_avg_latency
64-bit expansion:total_ops
64-bit expansion:sys_avg_latency

36 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
SPC-1 benchmark: Average latency and throughput
During in-place expansion

Initial short spike in latency

Expansion process starts

Baseline:sys_avg_latency
64-bit expansion:sys_avg_latency Baseline:total_ops
64-bit expansion:total_ops

37 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Sample completion times
Various configurations and in-place expansion completion times

Usage on Each Completion Time

Aggregate Size Volume Size # of Volumes
Volume (Hours)
16TB 10GB 499 50% ~30

16TB 13TB 1 100% ~28

13TB 12TB 1 30% ~10

13TB 1TB 10 90% ~1

13TB 100GB 100 70% ~1

13TB 25GB 499 66% ~1

13TB 400GB 40 30% ~1

38 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Data ONTAP aggregates
References

39 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Resources
32-bit and 64-bit aggregates

TR-3978: 64-bit Aggregates Overview and Best Practices

32-Bit to 64-Bit In-Place Aggregate Expansion Technical FAQ
TR-3505: NetApp Deduplication for FAS Deployment and Implementation Guide
Data ONTAP 8.3 Logical Storage and Management Guide
Data ONTAP 8 64-Bit Aggregate Technical Presentation
TR-4052: Clustered Data ONTAP Transition Guide
Data Protection Online Backup and Recovery Guide for 7-Mode

40 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use
Thank You
Please send feedback to gvg@netapp.com

41 2015 NetApp, Inc. All rights reserved. NetApp Proprietary Limited Use