Академический Документы
Профессиональный Документы
Культура Документы
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Solution configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Configuration overview . . . . . . . . . . . . . . . . . . . . . . . . 4
Network configuration . . . . . . . . . . . . . . . . . . . . . . . . . 4
Server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Mailbox server . . . . . . . . . . . . . . . . . . . . . . . . . . 6
LoadGen servers . . . . . . . . . . . . . . . . . . . . . . . . . 8
Naming convention . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Storage configuration . . . . . . . . . . . . . . . . . . . . . . . . . 10
Storage planning . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Test objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Functional testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Database seeding . . . . . . . . . . . . . . . . . . . . . . . . 14
Performance testing . . . . . . . . . . . . . . . . . . . . . . . . . . 17
LoadGen runs . . . . . . . . . . . . . . . . . . . . . . . . . . 17
HT delivery queues . . . . . . . . . . . . . . . . . . . . . . 33
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
EVA8100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Script reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
EVA configuration . . . . . . . . . . . . . . . . . . . . . . . . 41
Overview
Customers routinely request assistance from HP in validating the use of Microsoft Server Exchange
2007 and its new features, such as Cluster Continuous Replication (CCR), before they implement it
into production.
For this proof of concept (POC) project, HP replicated 28,000 user mailboxes in an Exchange 2007
CCR configuration. This quantity is chosen by calculating the number of mailboxes that could be
supported on a single Enterprise Virtual Array (EVA) 8100, while allowing a portion of disks on
each EVA for fibre-attached ATA (FATA) drives to store disk-based backups. This design also allows
for EVA growth by adding drives. The EVA 8100 is flexible and is capable of supporting different
configurations based on different design criteria. The EVA configuration used fits the requirements of
the design for this specific project.
A second EVA8100 and seven Exchange 2007 servers are used to host the CCR passive nodes of
the mailbox servers.
In this configuration, Exchange 2007 CCR nodes are deployed within the same datacenter. This
allows for failover to a secondary node, with a separate copy of Exchange 2007 database data,
in the event of an active cluster node failure.
An interesting feature of Exchange 2007 CCR is its ability to implement Exchange backups on the
passive node of the CCR cluster. This reduces the performance impact on the active node during
backup operations, which allows for a longer backup window. Backup is not tested as part of
this project.
3
Solution configuration
Configuration overview
The general configuration represents a single datacenter site, which depicts the most likely
configuration for CCR deployment.
Network configuration
For this proof of concept, the network configuration consists of HP ProCurve 2900 48G 48-port
Gigabit Ethernet switches placed into the server racks. This particular switch platform is chosen to
provide VLAN capability for network traffic segmentation between Exchange CCR clusters, should
this functionality be needed for the required performance level.
4
Figure 2. Server hardware configuration - Physical
DNS is implemented on all domain controllers and the DNS zones are configured as Active Directory
integrated with both forward and reverse lookup zones.
Server configuration
This design determines the proper server distribution and sizing, and the storage configuration. The
infrastructure component configurations have been maintained as closely as possible with Microsoft
best practices to ensure a valid proof of concept. The configurations for Exchange 2007 CCR
represent a calculated approximation and are validated throughout testing. The configurations are
based on the use of the Microsoft Storage Planning Calculator and the HP Exchange 2007 Calculator.
5
Mailbox server
In this design, three LUN presentation methodologies are used in the configuration to compare
administrative usability:
• Multi-root mount point with one anchor mount point per database and log LUN pair
• Dual-root mount point with one anchor mount point for all database LUNs and another anchor
mount point for all log LUNs
6
Table 3. Multi-root mount point configuration
F: SG01-07_MP
G: SG08-14_MP
H: SG15-21_MP
I: SG22-27_MP
The hub transport (HT) and client access servers (CAS) are combined in this implementation. The
CAS server provides Outlook clients with access to the availability service and auto-discovery service,
and also serves as the web-based distribution point for the offline address book (OAB).
The file share witness (FSW) for the mailbox cluster servers is placed on one of the
HT/CAS servers per Microsoft’s best practice recommendation. For more information, see
http://technet.microsoft.com/en-us/library/bb123996.aspx.
7
Table 5. Local storage - HT/CAS server
Z: DVD-ROM
Z: DVD-ROM
LoadGen servers
Z: DVD-ROM
The LoadGen configuration consists of a master server that controls the LoadGen environment, and
up to eight remote servers that generate the load against the Exchange environment.
An average Outlook 2007 cached-mode workload—10 messages sent and 40 received per day—is
targeted for use in load generation with an average message size of 104 KB. Each mailbox is
initialized with approximately 205 MB of data.
LoadGen generates a steady workload across the entire simulated day and does not generate
workload peaks that are seen in a production environment. By contrast, the Microsoft Storage
Planning Calculator used to size this configuration generates sizing based on the peak two hours of
the day—when it is expected that the majority of users will log in, read and reply to their e-mail
8
messages, and send new messages. This creates a difference in the type of work the mailbox servers
are performing in the test environment, which results in a lower, more even load when using LoadGen.
Naming convention
Using a well-thought-out naming convention is critical for operational efficiency. It allows rapid
identification of server names, server roles, databases, and storage group names. The consistent
use of a sound naming convention allows for more rapid development of a script library to enable
accurate and repeatable Exchange administration.
The following are the naming conventions used for objects in this configuration.
General server:
Mailbox server:
Storage group/database:
MB1 denotes Mailbox Cluster 1, SG01 denotes Storage Group 01, and DB01 denotes
Database 01.
9
The naming convention can be expanded to include additional information for easy sorting of
servers by location and function. The following example shows a site name that is added to the
beginning of the server name.
The advantage of this naming convention is that objects will first sort by site groups, then by object
type groups, and finally by uniqueness number.
Storage configuration
The HP StorageWorks Enterprise Virtual Array (EVA) 8100 platform is used to host all Exchange
2007 SAN storage.
HP StorageWorks 4/32B SAN switches are used for Fibre Channel (FC) connectivity between
the server host bus adapters (HBAs) and the EVA. The switches are deployed in a stand-alone
configuration with two switches shared per EVA. Use of the HP MPIO full-featured DSM for
EVA4x00/6x00/8x00 families of disk arrays permit multi-path I/O from the servers to the EVA for
fault tolerance.
LUNs on the EVA are balanced across both controllers with both the database and log LUNs for a
particular set of Exchange storage groups hosted on the same controller.
Note
In this testing, it is important to understand the performance and capacity profiles for the user mailboxes to calculate the
appropriate storage requirements. Since this is the testing objective, the tables below represent the best guess for this
unique environment.
Storage planning
The Microsoft Exchange 2007 Storage Calculator v9.4 and the HP Storage Planning Calculator
are used to generate the figures for this configuration. The configuration is based on a maximum
10
database size of 40 GB, not including overhead. Both performance and capacity are analyzed to
derive the tested configuration. For links to these calculators, see For more information.
148 39 GB 47 GB 19 GB
Note
It is important to consider factors such as database white space, deleted items dumpster, content indexing, and maintenance
in the total storage provisioning.
Best practice
EVA best practice is to have a number of disks divisible by eight. Formatted capacity of the 300 GB drive is ~ 279 GB.
11
Testing
Test objectives
The objectives for this project include:
Functional validation centers on simulating common equipment failure scenarios to ensure the impact
of implementing CCR will not adversely affect the Outlook user’s experience. Functional testing is
performed while running LoadGen to simulate Outlook client load and to closely simulate production
failures.
• Verify that acceptable server and storage performance occurs using the tested configuration.
• Verify that the failover and failback process works as expected under the following scenarios:
Functional testing
Functional testing is performed to ensure the Exchange configuration is in a healthy state and is able
to operate as expected. Functional testing is also used to perform common operations required when
running an Exchange CCR configuration and to simulate failures while observing the results of
the failure.
Microsoft recommends using the Exchange Best Practice Analyzer (ExBPA) to generate a report
depicting the health of the Exchange environment. ExBPA is included in the Exchange Management
Console and is located in the toolbox. ExBPA automatically updates when running to ensure the most
recent Microsoft best practices published are incorporated into the tool.
HP recommends that you run ExBPA when initially setting up the Exchange environment and run ExBPA
periodically, or when changes are made to Exchange, to make sure the configuration is healthy.
12
• Informational: Outlook connection range.
This is expected since all versions of Outlook are permitted in this configuration.
The expected value is 60,000. However, Exchange 2007 SP1 sets this to a higher value of
65,000 by default.
Exchange Profile Analyzer (ExPA) analyzes a set of Exchange servers and provides statistical
information regarding the distribution of messages. ExPA allows the detailed analysis of the size
and quantity of messages at the server level, database level, and individual mailbox level, which
identifies the distribution of messages in an Exchange environment.
Figure 4 depicts the results captured by running ExPA against one of the CCR databases.
13
Figure 4. Exchange Profile Analyzer results
The calculation below gives the rough approximation of the average message size of 72.34 KB.
The discrepancy between the 104 KB original average message size and the tested value is due to
the newer LoadGen build and the reduction in average message size from the older version to the
newer version. The newer versions of LoadGen are used at the recommendation of the Microsoft
LoadGen team during troubleshooting of LoadGen.
Database seeding
The Exchange CCR database seeding operation is responsible for the initial creation of data on the
passive node of the CCR cluster. The seeding operation is also performed when the database on
the passive node is out of date with the active node. This out-of-date condition may be caused by
several events, such as the prolonged unavailability of a server node or corrupt log files in the
passive node log sequence.
Testing is performed to determine the effect of the use of Cluster Administrator to manipulate the
passive node Windows cluster resources:
1. The passive node is evicted from the Windows cluster using Cluster Administrator.
14
2. The database and log data are deleted on the passive node.
3. The passive node is joined back into the Windows cluster using Cluster Administrator.
4. The data from the active CCR node is replicated to the passive CCR node.
Run the Update-StorageGroupCopy Cmdlet from the passive node, or copy the database
from the active CCR node to the passive CCR node.
During the seeding process, which is used to move data to the passive CCR node, we discover that
the passive node is unable to perform the reseed operation. The following procedure is performed to
permit successful seeding of the passive CCR node.
2. Ensure the file locations for database and transaction logs are empty.
4. Reboot.
6. Reboot.
7. Turn off Microsoft Exchange Search Indexer service to reduce load while seeding.
9. Update storage group copy from the passive node for all storage groups.
12. Move the Exchange and Windows cluster resources to the passive node to verify that all
functionality is restored.
Once the steps above are completed, seeding of the databases has successfully completed.
Each node of the Exchange CCR cluster pair is connected to a separate EVA8100, unlike single
copy clusters (SCC) that use shared storage. Due to the use of CCR, two copies of the Exchange
database and log files exist. This provides the advantage of an up-to-date copy of the Exchange
data for recovery.
Testing is performed to determine if the loss of the dedicated Exchange storage from the active node
will adversely affect the availability of Exchange services to Outlook clients. This testing is performed
while LoadGen is running against three of the mailbox clusters.
The following test cases have similar results and are grouped together for convenience:
15
Figure 5. Storage or storage connectivity loss
Outcome
• Windows cluster resources (IP, cluster name and quorum) stay online.
This is expected.
• Exchange cluster resources (IP, network name, system attendant, and information store) stay online.
This is expected.
This is expected.
Impact
• It is expected that the loss of storage on the active node will take the Exchange databases offline.
This will have a significant impact to the users as they will lose connectivity to their mailboxes.
This is due to the fact that there are no cluster disk resources when using Exchange CCR.
• In production, the use of E2K7 CCR will require monitoring of database states using a tool, such
as Microsoft Operations Manager, to alert the operations team that the Exchange databases are
in a dismounted state and require manual intervention.
Each Exchange CCR cluster node is configured with public and private network connections. The
public network is used by default for client and replication traffic, while the private network is used
for heartbeat traffic.
Testing is performed to determine if the loss of network connectivity by the active node will adversely
affect Exchange services availability to Outlook clients. This test is performed while LoadGen is
running against three of the mailbox clusters.
The following cases had similar results and are grouped together for convenience:
16
• Loss of network connectivity at the server
Outcome
• Windows cluster resources (IP, cluster name, and quorum) failover to the passive node.
This is expected.
• Exchange cluster resources (IP, network name, system attendant, and information store) failover
to the passive node.
This is expected.
• Exchange database resources failover to the passive node and are mounted.
This is expected.
Impact
• In production, the use of E2K7 CCR will require monitoring of database states using a tool, such
as Microsoft Operations Manager, to alert the operations team in the event that any Exchange
database is in a dismounted state. This condition can occur after the automated failover to
the passive node and will require manual intervention. The ability to automatically mount
databases is affected by the AutoDatabaseMountDial attribute. This attribute can be set using
the Set-MailboxServer Cmdlet to one of three values: Lossless, Good Availability, and
Best Availability. Each of these settings determines the quantity of transaction logs that can be
missing, and the database still mounts successfully.
Performance testing
LoadGen runs
In the course of LoadGen testing, Microsoft was consulted due to concerns about using LoadGen
to generate a stable test run. Subsequent to this, multiple versions of LoadGen were built by the
17
Microsoft LoadGen team to improve the way LoadGen handles the interaction between the master
server and the LoadGen remote clients. LoadGen version 08.02.0008 was used for this testing.
Throughout the debugging process, as the LoadGen builds were changed and provided to us,
the functionality of LoadGen changed, resulting in unexpectedly low per user IOPS. Subsequent
investigation with Microsoft reveals that this is a function of the reduction of the LoadGen default
average message size from 50 KB to 32 KB, and the reduction of the type of work being performed
by LoadGen. The reduction in I/O is revealed when analyzing the IOPS per user from test runs and
by working with Microsoft through additional debugging runs. For the specific testing performed
during this project, Microsoft recommended the use of the LoadGen online-heavy profile (20 sent/80
received) for subsequent test runs to increase the IOPS generated.
Based on Microsoft’s recommendation, LoadGen load balancing is not used with the remote load
generator configuration for this testing.
Initialization runs
• LoadGen is run multiple times to initialize the test environment due to the following:
– One run is required to create each of the seven server object organizational units (OUs) in
Active Directory (AD) during LoadGen topology creation. The objects are then populated
into the created OU. Each subsequent run creates the next server object OU and allows the
creation of objects into that OU. This issue has been reported to Microsoft.
– Initialization is run multiple times as new LoadGen buddy builds are released to address
issues experienced.
Debugging runs
• The Microsoft LoadGen team spent many cycles performing debugging in this POC environment
to address LoadGen issues, including the use of remote LoadGen servers.
• The LoadGen XML file and configuration used for debugging that specified Outlook 2007 online
heavy mode is used for subsequent testing.
• Testing reveals that LoadGen tasks are being queued across all of the LoadGen remote servers,
and the Microsoft LoadGen developer recommends the reconfiguration of network resources. As
part of this reconfiguration, the Scalable Network Pack TCP Chimney Offload feature in Windows
2003 R2 (and above) is disabled by running the following command:
This results in the LoadGen task queues dropping as the tasks are processed until the queues are
empty. This behavior is consistent on all LoadGen remote servers. Microsoft has since released a
critical update for Windows Server 2003 that disables this feature. For more information, see
http://support.microsoft.com/?kbid=948496.
• The performance capture runs were completed with LoadGen in the post-debugging configuration
without any additional initializations or baselines being performed. The profile used is Outlook
2007 Online-Heavy per Microsoft recommendation.
• The LoadGen runs are configured for an eight-hour simulated day with a ten hour run time.
18
• Performance capture is only started after the LoadGen test engines are started on the LoadGen
master server. This is a requirement to successfully capture LoadGen engine performance data.
Performance capture runs are configured to automatically stop after ten hours.
• A steady-state time period is chosen from the middle of the test run for analysis to ensure that
LoadGen events, such as client logon and Exchange online maintenance, are not analyzed.
• Data represented for comparison is from the same database LUNs across all tests. Data from
multiple LUNs and counters are reviewed, and a representative sample is included in this
document.
• LoadGen does not perform an identical workload on each server, and some variation is seen
in the data reported.
Testing for LoadGen Run 080212 is performed using LoadGen Outlook 2007 Online-Heavy.
LoadGen did not generate a high enough load to be able to extrapolate a normal user load. The
normal user load expected from LoadGen testing using the Outlook 2007 Online-Heavy profile is
0.40 IOPS or above.
Figure 7 depicts the disk transfers per second on database LUNs for the entire duration of LG
Run 080212. The data shows the initial LoadGen logon and ramp up, and the start of online
maintenance.
19
Figure 7. LG Run 080212 - database LUN - overview
The data contained in Figure 7 is only used to determine the steady-state time period used for further
analysis of this test run (1900 hours to 2200 hours).
The expected client workload for a LoadGen Default Outlook 2007 Online-Heavy profile is 20
sent/80 received.
Figure 8 shows the actual workload generated during the test standardized to an eight-hour day. The
following calculation is used to adjust for the three-hour, steady-state window.
Calculation:
The average sent/received per user per day across all servers is shown in Figure 8.
20
Figure 8. LG Run 080212 - Message send and receive statistics
Figure 9, Figure 10, and Figure 11 depict the disk I/O on the first database LUN on each of the
three Exchange 2007 CCR clusters configured for the LG Run 080212.
Modification is made to reduce the quantity of memory on two servers to allow the simultaneous
testing of the effect of memory. It is important to reiterate that LoadGen averages work evenly across
the entire day, while actual production workload will reveal peaks and valleys.
• MB2N1 − 16 GB RAM
• MB3N1 − 12 GB RAM
• MB4N1 − 8 GB RAM
21
Figure 9. LG Run 080212 - MB2N1 - 16 GB RAM
22
Figure 10. LG Run 080212 - MB3N1 - 12 GB RAM
23
Figure 11. LG Run 080212 - MB4N1 - 8 GB RAM
In this test, additional memory reduces the I/O requirement for a mailbox server. The most dramatic
effect is achieved when comparing the 16 GB (0.074 IOPS per user) and 12 GB (0.155 IOPS per
user) configurations, which reflects twice the database I/O with the reduction of memory.
Figure 12 depicts the processor use on each of the three Exchange 2007 CCR clusters configured
for the LG Run 080212. The server memory configuration ranges from 16 GB to 8 GB and is
denoted on the charts.
24
Figure 12. LG Run 080212 - % processor time
Note
The % processor time value is well below the MS recommendation of less than 90 (at all times).
There is a small impact on CPU use with the change in the amount of RAM. There is sufficient
processor headroom to support the use of third-party applications, such as anti-virus scanners.
Figure 13 depicts the RPC averaged latency on each of the three Exchange 2007 CCR clusters
configured for the LG Run 080212. The server memory configuration ranges from 16 GB to 8 GB
and is denoted on the charts.
25
Figure 13. LG Run 080212 - RPC averaged latency
Note
RPC averaged latency is well below the MS recommendation of less than 50 ms.
In this test, additional memory positively affects the RPC averaged latency on the mailbox server. At
no time does the RPC averaged latency approach or exceed the Microsoft-recommended threshold of
50 ms. The effect of database cache is apparent when comparing the 8 GB and 16 GB data. Client
requests are most efficiently served from cache and not from disk.
Figure 14 depicts the number of RPC requests outstanding on each of the three Exchange 2007 CCR
clusters configured for the LG Run 080212. The server memory configuration ranges from 16 GB to
8 GB and is denoted on the charts.
26
Figure 14. LG Run 080212 - RPC requests
Note
RPC requests is well below the MS recommendation of less than 20.
In this test, additional memory positively affects the mailbox server’s ability to handle RPC requests.
At no time does RPC requests approach or exceed the Microsoft recommended threshold of 20 for
any of the configured servers.
Figure 15 depicts the database cache size on each of the three Exchange 2007 CCR clusters
configured for the LG Run 080212. The server memory configuration ranges from 16 GB to 8 GB
and is denoted on the chart.
27
Figure 15. LG Run 080212 – database cache size
In this test, Exchange server uses additional memory for the database cache. Exchange uses
available memory for database cache if it is available and is needed.
With 16 GB of RAM on MB2N1, Exchange is able to reduce cache over time as it no longer needs
all of the cache it allocated. This indicates that sufficient memory is available for cache, given the
LoadGen heavy profile we are using.
Figure 16 depicts the quantity of messages queued for delivery across the five hub transport (HT)
servers configured for the LG Run 080212.
28
Figure 16. LG Run 080212 - hub transport messages queued for delivery
HT servers are working efficiently and are not queuing messages, resulting in a bottleneck. For
the duration of the testing, there is an average of one message queued on each of the five hub
transport servers.
LoadGen profile
• 20 sent/48 received
IOPS generated
For LoadGen Run 080212, there are no significant bottlenecks, and the configuration performed
well. The effect of RAM is apparent in the IOPS generated during the LoadGen run. When using half
of the amount of RAM that the Storage Planning Calculator recommends, the IOPS double.
29
LoadGen in stress mode enables the generation of tasks as fast as the receiving Exchange server
can accept them. This mode eliminates the wait time between tasks that occurs when running in
normal mode. LoadGen stress mode allows the identification of bottlenecks across multiple server
and storage subsystems. The use of stress mode intentionally overloads the Exchange system; the
expected outcome is that the servers will not be able to handle the load generated.
LoadGen stress mode causes each LoadGen remote server to log on to each mailbox; this results in
each mailbox processing multiple, simultaneous client connections.
Figure 17 depicts the disk transfers per second on each of the three Exchange 2007 CCR clusters
configured for the LG Run 080214.
The server memory configuration for the stress mode LG Run 080214 is as follows:
• MB2N1 − 8 GB RAM
• MB3N1 − 16 GB RAM
• MB4N1 − 12 GB RAM
Note
The server memory configuration is changed in earlier testing to ensure that the effect of memory on performance is consistent
when the quantity of RAM is changed. For example, the performance on a server with 8 GB of RAM is consistent with the
performance on another server when it is configured with 8 GB of RAM.
30
Note
Client logons do not complete until approximately 1630 hours.
When LoadGen is configured to utilize stress mode, the following approximate per user IOPS are
reached:
These results indicate that LoadGen running in stress mode is able to generate a significant load on
the mailbox servers.
Figure 18 depicts the processor use on each of the three Exchange 2007 CCR clusters configured for
the LG Run 080214.
Note
Client logons did not complete until approximately 1,630 hours.
31
Due to the significant load that LoadGen stress mode generates, % processor time is above the MS
recommendation of less than 90. This is a typical outcome of stress mode, where the tool will run
until all server resources are utilized.
Figure 19 depicts the RPC averaged latency on each of the three Exchange 2007 CCR clusters
configured for the LG Run 080214.
Note
Client logons do not complete until approximately 1,630 hours.
Due to the significant load that LoadGen stress mode generates, RPC averaged latency is well above
the MS recommendation of 50 ms. This is expected when using stress mode and indicates the
Outlook clients are experiencing delays when performing tasks against the Exchange server.
Figure 20 depicts the RPC requests outstanding on each of the three Exchange 2007 CCR clusters
configured for the LG Run 080214.
32
Figure 20. LG Run 080214 - RPC requests
Note
Client logons do not complete until approximately 1630 hours.
Due to the significant load that LoadGen stress mode generates, RPC requests are well above the MS
recommendation of 20. This is expected when using stress mode and indicates the Outlook clients
are experiencing delays when performing tasks against the Exchange server.
HT delivery queues
Figure 21 depicts the quantity of messages queued for delivery across the five HT servers configured
for the LG Run 080214.
33
Figure 21. LG Run 080214 - hub transport messages queued for delivery
Note
Client logons did not complete until approximately 1630 hours.
Due to the significant load that LoadGen stress mode generates, there are approximately 125,000
messages queued for delivery across the five HT servers at the completion of the test run. The
backlog of messages waiting on the HT server to be processed is a result of the bottlenecks identified
previously in areas such as processor use.
LoadGen profile
IOPS generated
For LoadGen Run 080214, there are bottlenecks across the Exchange configuration. Stress mode
is used to determine if this build of LoadGen is capable of generating a significant load on the
Exchange infrastructure.
34
Conclusion
The Exchange 2007 CCR POC configuration allows the validation of the server and storage
configuration for 28,000 mailboxes.
• Exchange 2007 SP1 CCR functionality works as expected for most common failure scenarios.
Functional testing identifies potential areas of concern listed below, along with areas where Exchange
2007 CCR could result in unexpected outages if monitoring is not configured and functional.
Performance testing with a workload generated by LoadGen shows that this configuration is sufficient
to support 28,000 heavy users with 250-MB mailboxes with CCR enabled.
The use of Exchange 2007 CCR host-based replication can provide customers with a valuable
tool to enable the reduction of time needed for recovery of Exchange data. The use of Exchange
2007 CCR technology has the potential to favorably impact the recovery time objective (RTO) and
recovery point objective (RPO) by allowing up-to-date copies of Exchange data for recovery on
separate servers and storage.
The most significant finding from testing is the dismounting of databases upon loss of storage or
connectivity on the active CCR node. While this behavior is expected, this is an issue since the
Exchange cluster resources, such as the System Attendant and Information Store, report they are up
and available while all of the databases are dismounted. This results in the loss of connectivity by
Outlook clients. This state requires the use of monitoring to ensure database availability, since the
cluster resources will not mount automatically in the event that one or all of the databases become
dismounted.
There are several additional areas of concern when using Exchange CCR that must be taken into
account before a change of this magnitude is implemented into a production environment:
• Additional training of support staff is required to ensure that operation of the new Exchange
CCR features is understood.
• Windows 2003/2008 majority node set clustering with FSW mechanism needs to be understood
by support staff, since it operates differently from a shared quorum cluster configuration.
• Database availability monitoring must be implemented to ensure that offline databases are
rapidly identified.
• Troubleshooting methodology changes due to new Windows clustering and Exchange CCR
features.
• Manual failover to the passive node may be required. There is no guarantee that databases will
automatically failover and mount.
• Backup and recovery methodology changes when using Exchange CCR, which results in
modification of the procedures detailing the backup and recovery of Exchange data.
35
• New recovery documentation development is required.
• Revision of the existing service level agreement may be required due to the different nature of
Exchange 2007 CCR.
http://hpwebgen.com/Questions.aspx?id=12046&pass=41514
36
Appendix A Performance measurement
Performance counter objects
The following list shows the general performance objects that are captured for LoadGen performance
runs by server type.
\LogicalDisk\*
\Memory\*
\Network Interface(*)\*
\NTDS\*
\Paging File(*)\*
\Process(*)\*
\Processor(*)\
\Server\*
\Memory\*
\Network Interface(*)\*
\Paging File(*)\*
\Process(*)\*
\Processor(*)\
\Server\*
\Memory\*
\Network Interface(*)\*
\Paging File(*)\*
\Process(*)\*
\Processor(*)\
\Server\*
Mailbox servers
37
\LogicalDisk(*)\*
\Memory\*
\MSExchange Database(*)\*
\MSExchangeIS(*)\*
\MSExchangeIS Client(*)\*
\MSExchangeIS Mailbox(*)\*
\Network Interface(*)\*
\Paging File(*)\*
\Process(*)\*
\Processor(*)\
\Server\*
EVA storage
38
Appendix B Bill of materials
Servers
Quantity Part number Description Comments
39
EVA8100
440 AG425B 300 GB 15K rpm dual-port 2/4 Gb/s FC-AL 1-inch (2.54
cm) drive
Infrastructure
Quantity Part number Description Comments
Miscellaneous
Quantity Part number Description Comments
40
Appendix C Exchange 2007 script samples
Script reference
The following sample scripts are broken into functional sections by the type of work being performed
and, where applicable, scripts are provided for the three types of LUN configuration used in this
testing.
EVA configuration
EVA configuration scripts are generated using the HP Storage System Scripting Utility (SSSU) to
capture the active configuration on EVA01 after storage configuration is completed. The script is
then modified to reference EVA02 and is processed to configure the second EVA identically to
the first and present the LUNs to the servers.
Once the disks are presented to the servers and verification that the correct configuration of drives is
present, DiskPart.exe is used in conjunction with text files to select, align, and label partitions.
After the partition creation process is completed, partitions are formatted and labeled, and folders
are created for Exchange databases and log files.
This DiskPart input text file depends on the appropriate LUNs being presented in a consistent order.
The alignment of disk partitions is an optional task, since the performance gain on an EVA 8100 is
negligible.
REM Version 1
REM
REM
select disk 2
assign letter=F
select disk 3
assign letter=G
select disk 4
assign letter=H
41
select disk 5
assign letter=I
select disk 6
assign letter=J
select disk 7
assign letter=K
select disk 8
assign letter=L
select disk 9
assign letter=M
exit
This Format script formats the aligned partitions using the NTFS file system, sets a volume name
(MB1N1_SG01-07_DB) and uses a 64 KB block allocation size. A block allocation size of 64 KB
is recommended for database partitions. A block allocation size of 8 KB (default) or 64 KB is
suitable for log partitions.
REM Version 1
REM
This folder creation script creates the folders used to organize the Exchange database and log file
structures.
42
REM HP CFT POC
REM Version 1
REM
MD F:\SG01_DB
MD F:\SG02_DB
MD F:\SG03_DB
MD F:\SG04_DB
MD F:\SG05_DB
MD F:\SG06_DB
MD F:\SG07_DB
MD G:\SG01_Log
MD G:\SG02_Log
MD G:\SG03_Log
MD G:\SG04_Log
MD G:\SG05_Log
MD G:\SG06_Log
MD G:\SG07_Log
MD H:\SG08_DB
MD H:\SG09_DB
MD H:\SG10_DB
MD H:\SG11_DB
MD H:\SG12_DB
MD H:\SG13_DB
MD H:\SG14_DB
MD I:\SG08_Log
MD I:\SG10_Log
MD I:\SG11_Log
MD I:\SG12_Log
MD I:\SG13_Log
MD I:\SG14_Log
MD J:\SG15_DB
MD J:\SG16_DB
43
MD J:\SG17_DB
MD J:\SG18_DB
MD J:\SG19_DB
MD J:\SG20_DB
MD J:\SG21_DB
MD K:\SG15_Log
MD K:\SG16_Log
MD K:\SG17_Log
MD K:\SG18_Log
MD K:\SG19_Log
MD K:\SG20_Log
MD I:\SG09_Log
MD K:\SG21_Log
MD L:\SG22_DB
MD L:\SG23_DB
MD L:\SG24_DB
MD L:\SG25_DB
MD L:\SG26_DB
MD L:\SG27_DB
MD M:\SG022_Log
MD M:\SG023_Log
MD M:\SG024_Log
MD M:\SG025_Log
MD M:\SG026_Log
MD M:\SG027_Log
Once the storage provisioning steps have been completed, the Exchange storage groups and
databases can be created. This section of scripts performs the tasks necessary for the creation,
labeling, and configuration of Exchange objects for this testing.
This storage group provisioning script creates storage groups on an Exchange 2007 server after the
Exchange 2007 CCR nodes have been installed, licensed, and verified for health.
The first section creates the storage group, sets the log and system folder paths, and enables circular
logging. Circular logging should only be used in non-production test environments and is set here to
44
allow the LoadGen initialization process to complete without filling log LUNs and dismounting the
Exchange databases.
The second section creates the Exchange database and sets the file path for each database.
The third section mounts the databases that are created. The identity used in the script to refer to
the database uses a unique reference, FYDIBOHF23SPDLT, which looks like a GUID but is actually
a transcription of EXCHANGE12ROCKS—this was chosen by the Exchange development team
when they needed to generate this unique value. This value is the same across all Exchange
implementations.
The script examples presented are the actual scripts used to perform Exchange manipulations. There
are many ways to create PowerShell functionality, and some of the functions in these scripts can be
performed in a single line. An example is presented below.
The Cmdlet used to mount each database individually that must be repeated 27 times with one
line per database is:
Note
Script lines have been wrapped for representation in this document.
#
=======================================================================
#
=======================================================================
45
new-StorageGroup -Server ’MB1CCR’ -CircularLoggingEnabled:$true -Name
’SG02’ -LogFolderPath ’G:\SG02_Log’ -SystemFolderPath ’G:\SG02_Log’
46
new-StorageGroup -Server ’MB1CCR’ -CircularLoggingEnabled:$true -Name
’SG21’ -LogFolderPath ’K:\SG21_Log’ -SystemFolderPath ’K:\SG21_Log’
47
new-mailboxdatabase -StorageGroup ’MB1CCR\SG13’ -Name ’MB1SG13DB01’
-EdbFilePath ’H:\SG13_DB\MB1SG13DB01.edb’
48
(FYDIBOHF23SPDLT),CN=Administrative Groups,CN=CFTOrg,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=cft,DC=lab’
49
mount-database -Identity ’CN=MB1SG14DB01,CN=SG14,CN=Informa
tionStore,CN=MB1CCR,CN=Servers,CN=Exchange Administrative Group
(FYDIBOHF23SPDLT),CN=Administrative Groups,CN=CFTOrg,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=cft,DC=lab’
50
(FYDIBOHF23SPDLT),CN=Administrative Groups,CN=CFTOrg,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=cft,DC=lab’
This script toggles the value to disable circular logging for all storage groups on a particular server
and restarts the MSExcahngeIS service to enable the new setting.
=======================================================================
# NAME: MB1_DisableCircularLogging.ps1
=======================================================================
$val = 0
$val++
Restart-Service MSExchangeIS
51
This script toggles the value to enable circular logging for all storage groups on a particular server
and restarts the MSExcahngeIS service to enable the new setting. This script must be run from
the ACTIVE CCR node.
=======================================================================
# NAME: MB1_EnableCircularLogging.ps1
=======================================================================
$val = 0
$val++
Restart-Service MSExchangeIS
#
=======================================================================
# NAME: MB1_Dismount-Database_ALL.ps1
#
=======================================================================
52
Dismount-Database -Identity ’MB1CCR\MB1SG01DB01’ -confirm:$false
#
=======================================================================
53
# NAME: MB1_Mount-Database_ALL.ps1
#
=======================================================================
54
This script suspends CCR replication for each storage group in preparation for a manual reseed
procedure.
#
=======================================================================
# NAME: MB1_Suspend-StorageGroupCopy_ALL_SG.ps1
# COMMENT:
#
=======================================================================
55
Suspend-StorageGroupCopy -Identity ’MB1CCR\SG22’ -Confirm:$false
This script performs a manual reseed procedure after deleting any existing files and restarts the
storage group copy process. This script must be run from the PASSIVE CCR node.
#
=======================================================================
# NAME: MB1_Update-StorageGroupCopy_ALL_SG.ps1
#
=======================================================================
56
Update-StorageGroupCopy -Identity ’MB1CCR\SG09’ -Confirm:$false
-DeleteExistingFiles:$True -Force:$True
57
This script enables CCR replication for each storage group.
# =====================================================================
# NAME: MB1_Resume-StorageGroupCopy_ALL_SG.ps1
# COMMENT:
# =====================================================================
58
Resume-StorageGroupCopy -Identity ’MB1CCR\SG24’ -Confirm:$false
This script is used to manually copy the active Exchange databases as an option to reseeding the
passive node databases. It requires that replication be suspended, the databases be dismounted,
and that the target directories are empty.
REM
=====================================================================
REM
REM
REM
REM
REM Comment2: Suspend all SG Copy, Dismount all DB on Server, Clean all
REM
REM======================================================================
59
copy \\mb1n1\h$\SG13_DB\mb1SG13DB01.edb h:\SG13_DB\mb1SG13DB01.edb
This script enumerates all databases on a particular server and sets the online maintenance schedule.
# ======================================================================
# NAME: MB1_SetMaintenanceSchedule.ps1
# ======================================================================
$val = 0
60
$val++
3. Press Enter.
Confirm that the Show Advanced Counters Item has been created in the appropriate location by
using RegEdit to view the object using PowerShell:
61
Appendix D Server build checklist
Server build consistency
The following items are completed on servers to ensure that all servers are consistently configured
and will be stable for the duration of the testing:
• PowerShell installed
• IIS installed
• HBAnywhere installed
• iLo named
62
• Application event log set to 40 MB in size
63
Appendix E EVA disk position
Physical EVA configuration
The following physical disk configuration is used for the proof of concept on all EVA8100s.
64
For more information
• HP Customer Focused Testing
http://www.hp.com/go/hpcft
• High availability for Microsoft Exchange Server 2007 SP1 using CCR with HP StorageWorks
EVA8100 (1-GB Mailbox)
http://h71028.www7.hp.com/ERC/downloads/4AA1-5675ENW.pdf
• Best practices for HP servers and HP Enterprise Virtual Array 8100 in a Microsoft Exchange
Server 2007 environment of 20,000 users
http://h71028.www7.hp.com/ERC/downloads/4AA1-5666ENW.pdf
http://msexchangeteam.com/archive/2007/01/15/432207.aspx
http://technet.microsoft.com/en-us/library/bb887628.aspx
http://technet.microsoft.com/en-us/exchange/bb330849.aspx
http://technet.microsoft.com/en-us/library/bb125177.aspx
http://blogs.technet.com/scottschnoll/archive/2006/10/06/Exchange-2007-_2D00_
Continuous-Replication-Architecture-and-Behavior.aspx
65