Вы находитесь на странице: 1из 39

The Host Server

Microsoft Windows Configuration Guide

May 2018

This guide provides configuration settings and considerations for Hosts running
Microsoft Windows.

Basic Microsoft Windows storage administration skills are assumed including how
to connect to iSCSI and Fibre Channel target ports and the discovering, mounting
and formatting of disk devices.

The Data Infrastructure Software Company


Table of contents

Changes made to this document 3


Microsoft Windows compatibility lists 4
Microsoft's own 'built-in' MPIO 4
DataCore Windows Integration Kit (WIK) 5
DataCore MPIO for Windows 6
Windows Offloaded Data Transfers (ODX) 8
Windows SCSI UNMAP 8
Microsoft's VSS Hardware Provider 9
DataCore's VSS Hardware Provider 9
The DataCore Server's settings 10
Operating System Type 10
Port roles 10
Multipathing support 10
ALUA support 11
Serving a Virtual Disks 11
The Windows Host's settings 13
Operating System Settings 13
Microsoft Cluster settings 14
Known issues 16
Affects Windows 2008, 2012 and 2016 Hosts 17
Affects Windows 2012 and 2016 Hosts 20
Affects Windows 2008 and 2012 Hosts 21
Appendix A 22
Preferred Server & Preferred Path settings 22
Appendix B 24
Reclaiming Storage from Disk Pools 24
Appendix C 27
How to configure Microsoft's own 'built-in' MPIO 27
How to verify MPIO support is configured correctly 30
Previous changes 35

Page | 2 The Host Server – Microsoft Windows Configuration Guide


Changes made to this document
The most recent version of this document is available from here:
http://datacore.custhelp.com/app/answers/detail/a_id/838

All changes since February 2018

Updated
The Windows Host Settings - Microsoft Cluster specific configuration settings
Two new subsections have been added:
 When using a File Share witness
 When using a Disk witness (including new important notes).

All previous changes


Please see page 35.

Page | 3 The Host Server – Microsoft Windows Configuration Guide


Microsoft Windows compatibility lists
Microsoft's own 'built-in' MPIO
SANsymphony

9.0 PSP 4 Update 4 (1) 10.0 (all versions)

Windows Version Without ALUA With ALUA Without ALUA With ALUA

XP / 7 / Vista
Not Supported Not Supported Not Supported Not Supported
8.x / 10

2000 / 2003 Not Supported Not Supported Not Supported Not Supported

2008 inc. R2 Not Supported Qualified (2) Not Supported Qualified (2)

2012 inc. R2 Not Supported Qualified (2) Not Supported Qualified (2)

2016 Not Supported Not Qualified Not Supported Qualified (2) (3)

Notes:
Qualified vs. Not Qualified vs. Not Supported
See page 7 for definitions.

DataCore Server Front-End Port connections


Fibre Channel connections are supported on all versions of Windows. ISCSI connections are
supported on all versions of Windows except Windows 2000 SP3.

How to configure Microsoft's own 'built-in' MPIO with SANsymphony


See Appendix C on page 27

1
SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now ‘End of Life’.
Please see: End of Life Notifications http://datacore.custhelp.com/app/answers/detail/a_id/1329
2
Only 'Round Robin with subset' or 'Failover only' MPIO Policies have been tested all other MPIO policies are
considered 'Not Qualified'.
3
Windows 2016 Hosts are supported with SANsymphony 10.0 PSP 6 or greater - earlier versions of SANsymphony
10.x are 'Not Qualified'.

Page | 4 The Host Server – Microsoft Windows Configuration Guide


Compatibility lists

DataCore Windows Integration Kit (WIK)


SANsymphony

9.0 PSP 4 Update 4 (1) 10.0 (all versions)

Windows Version Without ALUA With ALUA Without ALUA With ALUA

XP / 7 / Vista
Not Supported Not Supported Not Supported Not Supported
8.x / 10

2000 SP3 Not Supported Not Supported Not Supported Not Supported

2003 R2 Not Supported Not Supported Not Supported Not Supported

2008 inc. R2 (2) Qualified Qualified Qualified Qualified

2012 inc. R2 (3)(4) Qualified Qualified Qualified Qualified

2016 (5) Not Supported Not Supported Qualified Qualified

Notes:
Qualified vs. Not Qualified vs. Not Supported
See page 7 for definitions.

DataCore Server Front-End Port connections


Fibre Channel connections are supported on all versions of Windows. ISCSI connections are
supported on all versions of Windows except Windows 2000 SP3.

Host requirements when using DataCore's Windows Integration Kit with SANsymphony
Please refer to the Windows Integration Kit's own release notes.

1
SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now ‘End of Life’.
Please see: End of Life Notifications http://datacore.custhelp.com/app/answers/detail/a_id/1329
2
Hosts not running 'R2' versions (i.e. Windows 2008 or Windows 2008 SP2) must not use ALUA. However, ALUA
must be used when running on Windows 2008 R2.
3
Hosts running Windows 2012 'non-R2' using DataCore's Host Integration Kit 2.0 or earlier and must not have
ALUA enabled in the SANsymphony console..
4
Hosts running Windows 2012 R2 must use DataCore's Host Integration Kit 2.0+ and must have ALUA enabled in
the SANsymphony console.
5
Only use SANsymphony 10.0 PSP 6+ with DataCore's Windows Integration Kit 4.0+ and all hosts must have ALUA
enabled in the SANsymphony console.

Page | 5 The Host Server – Microsoft Windows Configuration Guide


Compatibility lists

DataCore MPIO for Windows


Important: This software is ‘End of Life’, see
http://datacore.custhelp.com/app/answers/detail/a_id/1329

DataCore MPIO is end of life and all versions of the Windows Operating System that this version
was qualified on are also End of life.

Please use the MPIO component from the 'DataCore Windows Integration Kit' software
package instead.

Page | 6 The Host Server – Microsoft Windows Configuration Guide


Compatibility lists

Qualified vs. Not Qualified vs. Not Supported

Qualified
This combination has been tested by DataCore and all the host-specific settings listed in this document applied
using non-mirrored, mirrored and Dual Virtual Disks.

Not Qualified
This combination has not yet been tested by DataCore using Mirrored or Dual Virtual Disks types. DataCore cannot
guarantee 'high availability' (failover/failback, continued access etc.) even if the host-specific settings listed in this
document are applied. Self-qualification may be possible please see Technical Support FAQ #1506

Mirrored or Dual Virtual Disks types are configured at the users own risk; however, any problems that are
encountered while using Windows versions that are 'Not Qualified' will still get root-cause analysis.

Non-mirrored Virtual Disks are always considered 'Qualified' - even for 'Not Qualified' combinations of
Windows/SANsymphony.

Not Supported
This combination has either failed 'high availability' testing by DataCore using Mirrored or Dual Virtual Disks types;
or the operating System's own requirements/limitations (e.g. age, specific hardware requirements) make it
impractical to test. DataCore will not guarantee 'high availability' (failover/failback, continued access etc.) if the
host-specific settings listed in this document are applied. Mirrored or Dual Virtual Disks types are configured at the
users own risk. Self-qualification is not possible.

Mirrored or Dual Virtual Disks types are configured at the users own risk; however, any problems that are
encountered while using Windows versions that are 'Not Supported' will get best-effort Technical Support (e.g. to
get access to Virtual Disks) but no root-cause analysis will be done.

Non-mirrored Virtual Disks are always considered 'Qualified' – even for 'Not Supported' combinations of
Windows/SANsymphony.

Windows versions that are End of Life (EOL)


For versions that are listed as 'Not Supported', self-qualification is not possible. For versions that are listed as 'Not
Qualified', self-qualification may be possible if there is an agreed ‘support contract’ with Microsoft as well. Please
contact DataCore Technical Support before attempting any self-qualification.

For any problems that are encountered while using Windows versions that are EOL with DataCore Software, only
best-effort Technical Support will be performed (e.g. to get access to Virtual Disks). Root-cause analysis will not be
done.

Non-mirrored Virtual Disks are always considered 'Qualified'.

Page | 7 The Host Server – Microsoft Windows Configuration Guide


Compatibility lists

Windows Offloaded Data Transfers (ODX)


SANsymphony

Windows
9.0 PSP 4 Update 4 (1) 10.0 (all versions)
Versions

2008 Tested/Works Tested/Works

2012 Tested/Works Tested/Works

2016 Not Supported Tested/Works

Windows SCSI UNMAP


SANsymphony

Windows
9.0 PSP 4 Update 4 (1) 10.0 (all versions)
Versions

2008 Does not Work Does not Work

2012 Tested/Works Tested/Works

2016 Not Tested Tested/Works

Notes:
Tested/Works vs Not Tested
This table only applies to combinations that are listed as 'Qualified' on page 5.

Off Load Data Transfers (ODX)


ODX will not work for for files smaller than 256Kb - See the 'Known Issues' section on page 16.
For EOL SANsymphony-V versions, ODX is not supported regardless of the Windows Operating
system and must be disabled on the Host. Please refer to http://technet.microsoft.com/en-
us/library/jj200627.aspx

Reclaiming storage from DataCore Disk Pools


See Appendix B: 'Reclaiming Storage from Disk Pools' on page 24 for version-specific 'how to'
instructions.

1
SANsymphony-V 8.x and all versions of SANsymphony 9.x before PSP4 Update 4 are now ‘End of Life’.
Please see: End of Life Notifications http://datacore.custhelp.com/app/answers/detail/a_id/1329

Page | 8 The Host Server – Microsoft Windows Configuration Guide


Compatibility lists

Microsoft's VSS Hardware Provider


SANsymphony

Windows
9.0 PSP 4 Update 4 (1) 10.0 (all versions)
Version

All Does not work Does not work

DataCore's VSS Hardware Provider


SANsymphony

Windows
9.0 PSP 4 Update 4 (1) 10.0 (all versions)
Version

XP / 7 / Vista /
Not Supported Not Supported
8.x / 10

2000 / 2003 Not Supported Not Supported

2008 inc. R2
Not Supported Not Supported
x86

2008 inc. R2
Qualified Qualified
x64

2012 inc. R2 Qualified (2) Qualified (3)

2016 Not Supported Qualified (4)

Notes:
Qualified vs. Not Qualified vs. Not Supported
See page 7 for definitions.

1
SANsymphony-V 8.x and all versions of SANsymphony-V 9.x before PSP4 Update 4 are now ‘End of Life’.
Please see: End of Life Notifications http://datacore.custhelp.com/app/answers/detail/a_id/1329
2
Use the Host Integration Kit 2.0 Update 1 with SANsymphony-V 9.0 PSP 4 Update 4 and Windows 2012/2012 R2.
Later versions of the Windows Integration Kit are not supported.
3
Use the Windows Integration Kit 3.0 or later with SANsymphony 10.x and Windows 2012/2012 R2. Earlier
versions of the Windows Integration Kit are not supported.
4
Use the Windows Integration Kit 4.0 or later with SANsymphony 10.x and Windows 2016. Earlier versions of the
Windows Integration Kit are not supported.

Page | 9 The Host Server – Microsoft Windows Configuration Guide


The DataCore Server's settings
Operating System Type
When registering the Host choose the appropriate menu option:

 Windows 2016
SANsymphony 10.0 PSP 6 or greater – 'Microsoft Windows 2012 and later'
Note: This option is only available from SANsymphony 10.0 PSP 6 or greater.

 Windows 2012 / 2012 R2


SANsymphony 10.0 PSP 6 or greater – 'Microsoft Windows 2012 and later'
Earlier versions of SANsymphony – 'Microsoft Windows 2012'

 All other Windows Operating systems


All versions of SANsymphony – 'Microsoft Windows'

Also see the Registering Hosts section from the SANsymphony Help:
http://www.datacore.com/SSV-Webhelp/Hosts.htm

Port roles
Ports used for serving Virtual Disks to Hosts should only have the Front End (FE) role enabled.
Mixing other Port Role types may cause unexpected results as Ports that only have the FE role
enabled will be turned off when the DataCore Server software is stopped (even if the physical
server remains running). This helps to guarantee that any Hosts do not still try to access FE
Ports, for any reason, once the DataCore Software is stopped but where the DataCore Server
remains running. Any Port with the Mirror and/or Back End role enabled do not shut off when
the DataCore Server software is stopped but still remain active.

Multipathing support
The Multipathing Support option should be enabled so that Mirrored Virtual Disks or Dual
Virtual Disks can be served to Hosts from all available DataCore FE ports. Also see the
Multipathing Support section from the SANsymphony Help: http://www.datacore.com/SSV-
Webhelp/Hosts.htm

Non-mirrored Virtual Disks and Multipathing


Non-mirrored Virtual Disks can still be served to multiple Hosts and/or multiple Host Ports from
one or more DataCore Server FE Ports if required; in this case the Host can use its own
multipathing software to manage the multiple Host paths to the Single Virtual Disk as if it was a
Mirrored or Dual Virtual Disk.
Note: Hosts that have non-mirrored Virtual Disks served to them do not need Multipathing
Support enabled unless they have other Mirrored or Dual Virtual Disks served as well.

Page | 10 The Host Server – Microsoft Windows Configuration Guide


The DataCore Server's settings

ALUA support
The ALUA support option (Asymmetrical Logical Unit Access) should be enabled if required and
if Multipathing Support has been also been enabled (see above). Please refer to the
compatibility lists on the previous pages for combinations of Host operating system, MPIO
products and SANsymphony versions that are known to work together with ALUA. More
information on 'Preferred Servers' and 'Preferred Paths' used by the ALUA function can be
found on in Appendix A on page 22.

Serving a Virtual Disks


Serving a Virtual Disk for the first time
DataCore recommends that before serving Virtual Disks for the first time to a Host, that all
DataCore Front-End ports on all DataCore Servers are correctly discovered by the Host first.
Then, from within the SANsymphony Console, the Virtual Disk is marked Online, up to date and
that the storage sources have a host access status of Read/Write.

Serving a Virtual Disk to more than one Host port


DataCore Virtual Disks always have their own unique Network Address Authority (NAA)
identifier that a Host can use to manage the same Virtual Disk being served to multiple Ports on
the same Host Server and the same Virtual Disk being served to multiple Hosts.

See the SCSI Standard Inquiry Data section from the online Help for more information on this:
http://www.datacore.com/SSV-Webhelp/Changing_Virtual_Disk_Settings.htm

While DataCore cannot guarantee that a disk device's NAA is used by a Host's operating system
to identify a disk device served to it over different paths generally we have found that it is. And
while there is sometimes a convention that all paths by the same disk device should always
using the same LUN 'number' to guarantees consistency for device identification, this may not
be technically true.

Always refer to the Host Operating System vendor’s own documentation for advice on this.

DataCore's Software does, however always try to create mappings between the Host's ports
and the DataCore Server's Front-end (FE) ports for a Virtual Disk using the same LUN number(1)
where it can. The software will first find the next available (lowest) LUN 'number' for the Host-
DataCore FE mapping combination being applied and will then try to apply that same LUN
number for all other mappings that are being attempted when the Virtual Disk is being served.
If any Host-DataCore FE port combination being requested at that moment is already using that
same LUN number (e.g. if a Host has other Virtual Disks served to it from previous) then the

1
The software will also try to match a LUN 'number' for all DataCore Server Mirror Port mappings of a Virtual Disk
too, although the Host does not 'see' these mirror mappings and so this does not technically need to be the same
as the Front End port mappings (or indeed as other Mirror Path mappings for the same Virtual Disk). Having Mirror
mappings using different LUNs has no functional impact on the Host or DataCore Server at all.

Page | 11 The Host Server – Microsoft Windows Configuration Guide


The DataCore Server's settings

software will find the next available LUN number and apply that to those specific Host-
DataCore FE mappings only.

Serving a Virtual Disks to a Microsoft Cluster


In the SANsymphony Management Console, define one Host for each Windows cluster node
(i.e. do not just create a ‘single’ Host for all cluster nodes and assign all the ports to that single
Host).

Only cluster node Hosts that have more than one Front End (FE) port connected to a DataCore
Server for any Virtual Disk require additional fail-over software in order to failover over
between FE ports on the same cluster node. See the 'MPIO compatibility lists' on page 4 for
combinations of Host operating system, MPIO products and SANsymphony versions that are
known to work together.

Page | 12 The Host Server – Microsoft Windows Configuration Guide


The Windows Host's settings
Operating System Settings
Windows Updates

DataCore strongly recommend apply the 'December 2016 Security Monthly Quality Rollup for
Windows 8.1 and Windows Server 2012 R2' on all Windows 2012 R2 DataCore Servers.

https://support.microsoft.com/en-us/help/3205401

There are a number of TCP/IP improvements including a fix for iSCSI Latency that DataCore had
identified and reported to Microsoft and was originally included in the ' November 15, 2016—
KB3197875 (Preview of Monthly Rollup)'.

Disk Timeout

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk

Modify the 'TimeOutValue' DWORD and give it a value of 3c (this is '60' seconds in Hex)

Page | 13 The Host Server – Microsoft Windows Configuration Guide


The Windows Host's settings

Microsoft Cluster specific configuration settings


When using a File Share witness
No additional changes are required.

When using a Disk witness

 Change the '$cluster .QuorumArbitrationTimeMax' from 20 (default) to 60 seconds

Run the following PowerShell commands on any host that is a member of the cluster:

1. $cluster = Get-Cluster
2. $cluster.QuorumArbitrationTimeMax=60
3. $cluster.QuorumArbitrationTimeMax

Here is an example showing the default value (20 seconds), changing it to 60 seconds and then
verifying the change.

Important notes
 This is a global setting that applies to all hosts in the cluster. No reboot is needed and is
effective immediately.

 Changing from a File Share witness to a Disk witness will set the
$cluster.QuorumArbitrationTimeMax value to the Disk witness default of 20
seconds. Re-run the commands above to put the setting back to 60 seconds.

 The setting maybe lost after rebooting the Cluster. Always verify the
$cluster.QuorumArbitrationTimeMax value after any reboot of the Host(s) and
re-run the commands above, if required, to put the setting back to 60 seconds.

Page | 14 The Host Server – Microsoft Windows Configuration Guide


The Windows Host's settings

When using the DataCore Windows Integration Kit’s MPIO software

DataCore's own MPIO solutions always rely on Microsoft's own MPIO system driver so it is
important that certain MPIO-related registry values are set to the following

The following is a list of expected default values:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\mpio\Parameters]

"PathVerifyEnabled"=dword:00000000
"PathVerificationPeriod"=dword:0000001e
"PDORemovePeriod"=dword:00000014
"DiskPathCheckDisabled"=dword:00000000
"DiskPathCheckInterval"=dword:0000000a

Always run the Cluster ‘Validate Configuration’ wizard

DataCore recommend that you run this wizard any time that new Virtual Disks are served to the
cluster but before they are used for production data just to confirm that they will function
correctly.

Page | 15 The Host Server – Microsoft Windows Configuration Guide


Known issues
The following is intended to make DataCore Software customers aware of any issues that may
affect performance, access or generally give unexpected results under certain conditions when
Windows Hosts are used with SANsymphony.

Some of the issues here have been found during DataCore’s own testing but many others are
issues reported by DataCore Software customers, where a specific problem had been identified
and then subsequently resolved.

DataCore cannot be held responsible for incorrect information regarding Microsoft products.
No assumption should be made that DataCore has direct communication with Microsoft
regarding the issues listed here and we always recommend that users contact their Microsoft
directly to see if there are any updates or fixes since they were reported to us.

Important notes:

The 'Known issues' listed in this document are for issues that affect Windows Hosts outside of
DataCore Software's control. For ‘Known issues’ for DataCore’s own Software products
including those that are specific to DataCore's Host Integration Kit, please refer to the relevant
DataCore Software Component’s release notes.

Unless otherwise stated, for the purpose of the Known Issues section:
Windows 2008 includes both the SP2 and 'non-SP2' versions
Windows 2012 includes both the R2 and 'non-R2' versions.

Page | 16 The Host Server – Microsoft Windows Configuration Guide


Affects Windows 2008, 2012 and 2016 Hosts
General operating system functions

Off Load Data Transfers (ODX) will not be used for for files smaller than 256Kb
File transfers for files smaller than 256 Kb will take place using 'traditional' copy methods so large numbers of
these small, files may impact Host performance for the duration of the transfer.
Also see:
http://msdn.microsoft.com/en-us/library/windows/hardware/dn265439%28v=vs.85%29.aspx

Fibre channel host bus adaptors

16 GB QLogic Fibre Channel Host Bus Adaptors


Do not use driver version 9.1.17.21 with firmware older than 08.03.00 when connecting to DataCore Server Front-
end ports using SANsymphony-V versions 10.0 PSP 4 Update 1 or earlier (including versions 9.0 PSP 4 Update 4 and
earlier) as in some cases this firmware on the Host can cause the DataCore Server to crash.

2 or 4 GB QLogic Fibre Channel Host Bus Adaptors


Do not use driver versions 9.0.1.12 or 9.0.2.11 as they may cause unexpected disconnections from the DataCore
Server.

Page | 17 The Host Server – Microsoft Windows Configuration Guide


Known issues - affects Windows 2008, 2102 and 2016 Hosts

Windows MPIO and DataCore MPIO (inc. DataCore WIK)

Microsoft's multipath bus driver (MPIO.sys) may not failover if two or more paths from a Host to a Virtual Disk
fail at the same time, or if the same Virtual Disk for two or more paths is unavailable.

This will affect Hosts using either DataCore's MPIO (e.g. the Windows Integration Kit) or Microsoft's own 'built-in'
MPIO - both of which rely on the MPIO.sys driver of the Windows operating system.

The Microsoft MPIO.sys driver builds its own list of all paths to any given disk device (e.g. a Virtual Disk) which also
includes the path that is designated as the 'next' path available for failover if the active path (e.g. the path that is
being used at that moment to send IO) reports an error either on the path or to the disk device itself. If an event,
such as a disk device or connection failure, cause both the active and the 'next' path (determined by the MPIO.sys
driver) to lose access to the same disk device, then no failover will occur even if there are other paths available to
that same disk device.

In this case the Host will lose access to the Virtual Disk!

Please apply one of the following fixes according to your Host’s Windows operating system:

For Windows 2016 Hosts - June 27, 2017—KB4022723 (OS Build 14393.1378)
https://support.microsoft.com/en-us/help/4022723

The specific fix is documented as "Addressed issue where you may lose access to storage disks when there are still
available paths if there is an error on one of the multipath I/O paths."

For Windows 2012 Hosts - July 11, 2017—KB4025336 (Monthly Rollup)


https://support.microsoft.com/en-us/help/4025336

This ‘July 11 2017’ KB includes everything from the June 27, 2017—KB4022720 (Preview …) where the specific fix is
documented as "Addressed an issue where MPIO failover stops after a disk has been surprise removed, identified by
Event ID 157: "Disk X has been surprised removed" when there are still viable paths to use. Scenario may occur
when the newly selected path belongs to the disk that has been surprised removed."

For any Windows 2008 or Windows 2012 (non-R2) Hosts:


No fix from Microsoft is planned for either of these versions of Windows Server.
As the problem is in Microsoft's own multipath bus driver, and because it is impossible for DataCore to know which
is going to be the 'next' designated active path at any given moment (as this is decided by the MPIO.sys driver) the
only safe configuration which will guarantee failover is when a mirrored Virtual Disk served to just one Host (FE)
port from each DataCore Server.

See figures on the following page for examples of configurations that are/are not supported.

Page | 18 The Host Server – Microsoft Windows Configuration Guide


Known issues - affects Windows 2008, 2102 and 2016 Hosts

Figure 1

Figure 2

Figure 3

Page | 19 The Host Server – Microsoft Windows Configuration Guide


Known issues - Affects Windows 2012 and 2016 Hosts

Affects Windows 2012 and 2016 Hosts


General operating system functions

Formatting a Virtual Disk may take longer than expected.

During the formatting of any filesystem, Windows will (by default) issue SCSI UNMAP write I/O for the entire size of
the filesystem. This will mean that the filesystem format process may take a long time to complete and will
generate significant write I/O to the Disk Pool regardless if the 'Perform a quick format' option is checked or not.
The recommendation from DataCore is to disable the 'SCSI TRIM and Unmap' feature on the Windows Host just for
the duration of the format.

From a CMD window on the Host run the fsutil command:

This will disable all UNMAP commands from the Host.


After the format has completed the UNMAP feature can be re-enabled by running the command:

The command takes effect immediately and without any impact on the Windows Host.
To verify the current setting for UNMAP at any time run the command:

DisableDeleteNotify=0 - indicates the UNMAP feature is enabled


DisableDeleteNotify=1 - indicates the UNMAP feature is disabled

Note: Disabling UNMAP will not affect the Windows ODX (offline data transfer) feature or vice versa.

Page | 20 The Host Server – Microsoft Windows Configuration Guide


Known issues - Affects Windows 2008 and 2012 Hosts

Affects Windows 2008 and 2012 Hosts


Microsoft Cluster Services (MSCS)

The Microsoft Cluster Validation test may report a warning message about an unsigned DataCore Multipath
driver (even though the driver is signed).
Microsoft has confirmed that in this case this message is benign and can be safely ignored and they will support
MSCS with the latest version of DataCore's Windows Integration Kit.

DataCore MPIO (inc. DataCore WIK)

Hosts using DataCore's Windows Integration Kit or DataCore's MPIO do not use the Preferred DataCore Server
immediately after a restart.
The host's operating system scans for the first available hardware path as it starts up and uses it regardless if the
path is on the preferred DataCore Server or not. The device path discovery is usually decided by the order of the
Host's PCI slots in the Host Server and is beyond the control of the DataCore MPIO software. Once all available
device paths have been detected by the host however, the Host will (eventually) move over to the path on the
'preferred' DataCore Server.

Page | 21 The Host Server – Microsoft Windows Configuration Guide


Appendix A
Preferred Server & Preferred Path settings
See the Preferred Servers and Preferred Paths sections from the SANsymphony Help:
http://www.datacore.com/SSV-Webhelp/Port_Connections_and_Paths.htm

Without ALUA enabled


If Hosts are registered without ALUA support, the Preferred Server and Preferred Path settings
will serve no function. All DataCore Servers and their respective Front End (FE) paths are
considered ‘equal’.

It is up to the Host’s own Operating System or Failover Software to determine which DataCore
Server is its preferred server.

With ALUA enabled


Setting the Preferred Server to ‘Auto’ (or an explicit DataCore Server), determines the DataCore
Server that is designated ‘Active Optimized’ for Host IO. The other DataCore Server is
designated ‘Active Non-Optimized’.

If for any reason the Storage Source on the preferred DataCore Server becomes unavailable,
and the Host Access for the Virtual Disk is set to Offline or Disabled, then the other DataCore
Server will be designated the ‘Active Optimized’ side. The Host will be notified by both
DataCore Servers that there has been an ALUA state change, forcing the Host to re-check the
ALUA state of both DataCore Servers and act accordingly.

If the Storage Source on the preferred DataCore Server becomes unavailable but the Host
Access for the Virtual Disk remains Read/Write, for example if only the Storage behind the
DataCore Server is unavailable but the FE and MR paths are all connected or if the Host
physically becomes disconnected from the preferred DataCore Server (e.g. Fibre Channel or
iSCSI Cable failure) then the ALUA state will not change for the remaining, ‘Active Non-
optimized’ side. However, in this case, the DataCore Server will not prevent access to the Host
nor will it change the way READ or WRITE IO is handled compared to the ‘Active Optimized’
side, but the Host will still register this DataCore Server’s Paths as ‘Active Non-Optimized’ which
may (or may not) affect how the Host behaves generally.

Page | 22 The Host Server – Microsoft Windows Configuration Guide


Appendix A – Preferred Server & Preferred Path settings

In the case where the Preferred Server is set to ‘All’, then both DataCore Servers are designated
‘Active Optimized’ for Host IO.

All IO requests from a Host will use all Paths to all DataCore Servers equally, regardless of the
distance that the IO has to travel to the DataCore Server. For this reason, the ‘All’ setting is not
normally recommended. If a Host has to send a WRITE IO to a ‘remote’ DataCore Server (where
the IO Path is significantly distant compared to the other ‘local’ DataCore Server), then the
WAIT times accrued by having to send the IO not only across the SAN to the remote DataCore
Server, but for the remote DataCore Server to mirror back to the local DataCore Server and
then for the mirror write to be acknowledged from the local DataCore Server to the remote
DataCore Server and finally for the acknowledgement to be sent to the Host back across the
SAN, can be significant.

The benefits of being able to use all Paths to all DataCore Servers for all Virtual Disks are not
always clear cut. Testing is advised.

For Preferred Path settings it is stated in the SANsymphony Help:


A preferred front-end path setting can also be set manually for a particular virtual disk. In this
case, the manual setting for a virtual disk overrides the preferred path created by the preferred
server setting for the host.

So for example, if the Preferred Server is designated as DataCore Server A and the Preferred
Paths are designated as DataCore Server B, then DataCore Server B will be the ‘Active
Optimized’ Side not DataCore Server A.

In a two-node Server group there is usually nothing to be gained by making the Preferred Path
setting different to the Preferred Server setting and it may also cause confusion when trying to
diagnose path problems, or when redesigning your DataCore SAN with regard to Host IO Paths.

For Server Groups that have three or more DataCore Servers, and where one (or more) of these
DataCore Servers shares Mirror Paths between other DataCore Servers setting the Preferred
Path makes more sense.

So for example, DataCore Server A has two mirrored Virtual Disks, one with DataCore Server B,
and one with DataCore Server C and DataCore Server B also has a mirrored Virtual Disk with
DataCore Server C then using just the Preferred Server setting to designate the ‘Active
Optimized’ side for the Host’s Virtual Disks becomes more complicated. In this case the
Preferred Path setting can be used to override the Preferred Server setting for a much more
granular level of control.

Page | 23 The Host Server – Microsoft Windows Configuration Guide


Appendix B
Reclaiming Storage from Disk Pools
Using SCSI UNMAP commands
Also known as 'SCSI TRIM and Unmap' is only qualified for Windows 2012 and Windows 2016
Hosts.

SANsymphony's Automatic Reclamation feature


DataCore Servers keep track of any 'all-zero' write I/O requests sent to Storage Allocation Units
(SAU) in all Disk Pools. When enough 'all-zero' writes have been detected to have been passed
down to an entire SAUs logical address space, that SAU will be immediately assigned as 'free'
(as if it had been manually reclaimed) and made available to the entire Disk Pool for future
(re)use.

No additional 'zeroing' of the Physical Disk or 'scanning' of the Disk Pool is required.

Important technical notes on Automatic Reclamation


The Disk Pool driver has a small amount of system memory that it uses keep a list of all address
spaces in a Disk Pool that are sent 'all-zero' writes; all other (non-zero) write requests are
ignored by the Automatic Reclamation feature and not included in the in-memory list.

Any all-zero write addresses that are detected to be physically 'adjacent' to each other – from a
block address point of view – the Disk Pool driver will 'merge' these requests together in the list
so as to keep the size of it as small as possible. Also as entire 'all-zeroed' SAUs are re-assigned
back to the Disk Pool, the record of all its address spaces is removed from the in-memory list
making space available for future all-zero writes to other SAUs that are still allocated.

However if write I/O pattern of the Hosts mean that the Disk Pool receives all-zero writes to
many, non-adjacent block addresses the list will require more space to keep track of them
compared to all-adjacent block addresses. In extreme cases, where the in-memory list can no
longer hold any more new all-zero writes (because all the allocated system memory for the
Automatic Reclamation feature has been used) the Disk Pool driver will discard the oldest
records of the all-zero writes to accommodate newer records of all-zero write I/O.

Likewise if a DataCore Server is rebooted for any reason, then the in-memory list is completely
lost and any knowledge of SAUs that were already partially detected as having been written
with all-zeroes will now no longer be remembered.

In both of these cases this can mean that, over time, even though technically an SAU may have
been completely overwritten with all-zero writes, the Disk Pool driver does not have a record

Page | 24 The Host Server – Microsoft Windows Configuration Guide


Appendix C – Reclaiming storage

that cover the entire address space of that SAU in its in-memory list and so the SAU will not be
made available to the Disk Pool but remain allocated to the Virtual Disk until any future all-zero
writes happen to re-write the same address spaces that were forgotten about previously by the
Disk Pool driver. In these scenarios, a Manual Reclamation will force the Disk Pool to re-read all
SAUs and perhaps detect those now missing all-zero address spaces.

See the section 'Manual Reclamation' on the next page for more information.

Reclaiming storage by sending all-zero writes to a Hosts's own


filesystem
For Windows Hosts that do not support SCSI UNMAP (or are using Virtual Disks served from
SANsymphony versions that do not yet support SCSI UNMAP) one suggestion would be to use
the 'sdelete' tool to write zeroes directly to the filesystem.

See the Windows Sysinternals - sdelete section from Microsoft's own website:
https://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

This I/O will then be detected by SANsymphony's Automatic Reclamation function (see previous
page for more details).

Also see: Performing Reclamation section from the SANsymphony Help:


http://www.datacore.com/SSV-Webhelp/Reclaiming_Virtual_Disk_Space.htm

SANsymphony's Manual Reclamation feature


Manual reclamation forces the Disk Pool driver to 'read' all SAUs currently assigned to a Virtual
Disk looking for SAUs that contain only all-zero IO data. Once detected, that SAU will be
immediately assigned as 'free' and made available to the entire Disk Pool for future (re)use.

No additional 'zeroing' of the Physical Disk is required.

Note that manual reclamation will create additional 'read' I/O on the Storage Array used by the
Disk Pool, as this process runs at 'low priority' it should not interfere with normal I/O
operations. However, caution is advised, especially when scripting the manual reclamation
process.

Manual Reclamation may still be required even when Automatic Reclamation has taken place
(see the 'Automatic Reclamation' section on the previous page for more information).

Page | 25 The Host Server – Microsoft Windows Configuration Guide


Appendix C – Reclaiming storage

How much storage will be reclaimed?


It is impossible to predict exactly how many Storage Allocation Units (SAUs) will be reclaimed.
For reclamation of an SAU to take place, it must contain only ‘all-zero’ block data over the
entire SAU else it will remain allocated and this is entirely dependent on how and where the
Host has written its data on the DataCore LUN.

For example, if the Host has written the data in such a way that every allocated SAU contains a
small amount of non-zero block data then no (or very few) SAUs can be reclaimed, even if the
total amount of data is much less than the total amount of assigned SAUs.

It may be possible to use the Host operating system’s own defragmentation tools to move and
any data that is spread out over the DataCore LUN so that it ends up as one or more large areas
of contiguous non-zero block addresses. This might then leave the the DataCore LUN with SAUs
that now only have all-zero data on them and that can then be reclaimed.

However care should be taken that the act of defragmenting the data itself does not cause
more SAU allocation as the block data is moved around (i.e. re-written to new areas on the
DataCore LUN) during the re-organization.

Page | 26 The Host Server – Microsoft Windows Configuration Guide


Appendix C
How to configure Microsoft's own 'built-in' MPIO
In order for SANsymphony Virtual Disks to be recognized as MPIO-capable when using
Microsoft's own 'built-in' MPIO feature the following settings are required.

A reboot of the Windows Host will be required, so plan accordingly.

1. On the Windows Host server, install the Multipath I/O Feature:

Normally a reboot of the Windows Host is not needed after installing the feature. However if
prompted by Windows (to reboot) then please reboot before continuing.

2. Open the 'Microsoft MultipathIO Configuration Tool' (mpiocpl.exe) and select the
'MPIO Devices' tab:

Page | 27 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

3. Click the 'Add' button

4. When the 'Add MPIO Support' window appears, enter the string "DataCoreVirtual
Disk"

Note: Please enter the string exactly as it appears above. The Vendor and Product IDs are case
sensitive and notice there is no space between the strings 'DataCore' and 'Virtual'.

It is also possible to use an Administrative command prompt to add the Device Hardware ID by
using the following 'mpclaim' command:

mpclaim –n –I –d "DataCoreVirtual Disk"

Page | 28 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

Important: If your current configuration was upgraded directly from SANsymphony version 7.x
or SANmelody version 3.x then the appropriate Vendor and Product IDs will need to be entered
as well.

For SANsymphony 7.x - enter the string "DataCoreSANsymphony"

mpclaim –n –I –d "DataCoreSANsymphony"

For SANmelody 3.x - enter the string "DataCoreSANmelody"

mpclaim –n –I –d "DataCoreSANmelody"

5. Click the OK button. Reboot when prompted.

Note: It is only possible to enter one Vendor/Product ID at a time. However, if you need to enter
either the SANsymphony 7 or SANmelody 3 Vendor/Product IDs as well, then do not reboot until
all additional entries have been added.

Page | 29 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

How to verify MPIO support is configured correctly


Method 1 - Using Device Manager
Open the 'Microsoft Multipath IO Configuration Tool' (mpiocpl.exe) and select the 'MPIO
Devices' tab. Check that the DataCore IDs you added in the previous steps are displayed.

The example below shows the MPIO Configuration Tool with all the possible DataCore
Vendor/Product IDs – including both SANsymphony 7 and SANmelody 3 strings.

Note: The 'Vendor 8Product 16' entry is a default Windows-specific entry and should be left
alone.

Page | 30 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

Verify in Device Manager that there is a 'DataCore Virtual Disk Multi-Path Disk Device' for each
Virtual Disk served to this Host from SANsymphony.

Note: There is no difference, in Device Manager, between how a mirrored and non-mirrored
DataCore Virtual Disk are displayed when MPIO has been configured.

If MPIO support has not been configured correctly then a 'DataCore Virtual Disk SCSI Disk
Device' disk drive will be displayed for each 'side' of a mirrored DataCore Virtual Disk.

An example of a single mirrored Virtual Disk served to a Host where MPIO is not installed
and/or has been configured incorrectly:

Page | 31 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

Method 2 - Using the MPIO_configuration.log file

Open the 'Microsoft MultipathIO Configuration Tool' (mpiocpl.exe) and select the
'Configuration Snapshot' tab. Check the 'Open File upon capture' box and then click the
'Capture' button.

The Log file will open and display the MPIO Disk information.

Page | 32 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

Method 3 - Using the mpclaim command


The same information as the 'Configuration Snapshot' tab from the previous example can be
extracted using the 'mpclaim' command.

For example:

mpclaim –s –d 1

Where the number at the end of the command refers to the MPIO Disk number.

Page | 33 The Host Server – Microsoft Windows Configuration Guide


Appendix D – How to configure Microsoft's own 'built-in' MPIO

Method 4 - Using the MPIO Disk Device's device manager property


Open Device Manager and locate a 'DataCore Virtual Disk Multi-Path Disk Device', right-click on
the Disk drive and select 'Properties' from the context menu.

Select the 'MPIO' tab,

Note: Only 'Round Robin with subset' or 'Failover only' MPIO Policies have been tested. Do not
change any other settings.

Page | 34 The Host Server – Microsoft Windows Configuration Guide


Previous changes
2018
February
Updated
General
This document has been reviewed for SANsymphony 10.0 PSP 7.
No additional settings or configurations are required.

Removed
Appendix B – Configuring Disk Pools
The information here has been removed as it is now superseded by the information in:
The DataCore Server- Best Practice Guidelines http://datacore.custhelp.com/app/answers/detail/a_id/1348
What was previously 'Appendix C' has now been moved to 'Appendix B' and so on.

2017
September
Updated
Known Issues
Windows MPIO and DataCore MPIO (including DataCore WIK).
Links for both the Windows Server 2016 and 2012 (R2-only) updates have been added and updated respectively.
The changes to the information are below:

Microsoft's multipath bus driver (MPIO.sys) may not failover if two or more paths from a Host to a Virtual Disk fail at the same time, or if the
same Virtual Disk for two or more paths is unavailable.

Windows 2012 (R2 only) and Windows 2016 Hosts: Microsoft have issued Windows Updates that fix this problem with their MPIO.sys driver.

Windows 2016 - June 27, 2017—KB4022723 (OS Build 14393.1378)


https://support.microsoft.com/en-us/help/4022723

The specific fix is documented as "Addressed issue where you may lose access to storage disks when there are still available paths if there is an
error on one of the multipath I/O paths."

Windows 2012 - July 11, 2017—KB4025336 (Monthly Rollup)


https://support.microsoft.com/en-us/help/4025336

This July 11 KB includes everything from the June 27, 2017—KB4022720 (Preview …) where the specific fix is documented as "Addressed an
issue where MPIO failover stops after a disk has been surprise removed, identified by Event ID 157: "Disk X has been surprised removed" when
there are still viable paths to use. Scenario may occur when the newly selected path belongs to the disk that has been surprised removed."

Windows 2008 and Windows 2012 (non-R2) Hosts: No fix from Microsoft is planned for either of these versions of Windows Server.

Page | 35 The Host Server – Microsoft Windows Configuration Guide


Previous changes
July
Updated
Known Issues
Windows MPIO and DataCore MPIO (including DataCore WIK)

Microsoft's multipath bus driver (MPIO.sys) may not failover if two or more paths from a Host to a Virtual Disk fail at the same time, or if the
same Virtual Disk for two or more paths is unavailable.

Windows 2012 and 2016 Hosts:


Microsoft are issuing a fix – see https://support.microsoft.com/en-us/help/4022720

"Addressed an issue where MPIO failover stops after a disk has been surprise removed, identified by Event ID 157: "Disk X has been surprised
removed" when there are still viable paths to use. Scenario may occur when the newly selected path belongs to the disk that has been surprised
removed."

Windows 2008 Hosts: No fix is planned. See the full Known Issues description in this document on how to configure Windows 2008 Hosts with
MPIO/WIK.

May
Added
Compatibility Lists - DataCore Windows Integration Kit (WIK)
Compatibility Lists – VSS Hardware Provider
Windows 2016 hosts are Qualified when using SANsymphony 10.0 PSP 6 or greater using DataCore's WIK 4.0 or
greater.

Appendix D - How to configure Microsoft's own 'built-in' MPIO


Information regarding how to set the correct Vendor/Product ID when using Microsoft's own MPIO, including
information on how to verify that MPIO has been set up correctly.

Updated
Compatibility Lists - DataCore MPIO for Windows
Only 'Round Robin with subset' or 'Failover only' MPIO Policies have been tested.

Removed
Compatibility Lists - DataCore MPIO for Windows
DataCore MPIO is end of life and all versions of the Windows Operating System that this version was qualified on
are also End of life. Therefore all the compatibility list information for this product has been removed.
Please use the MPIO component in the 'DataCore Windows Integration Kit' instead.

April
Added
Compatibility Lists
Windows Offloaded Data Transfers (ODX)
Windows SCSI UNMAP

The Windows Host's Settings – Microsoft Cluster Settings


Added instructions to set the Set '$cluster .QuorumArbitrationTimeMax' to 60 seconds

Known Issues - Affects Windows 2008, 2012 and 2016 Hosts


16 GB QLogic Fibre Channel Host Bus Adaptors
Do not use driver version 9.1.17.21 with firmware older than 08.03.00 when connecting to DataCore Server Front-end ports using
SANsymphony-V versions 10.0 PSP 4 Update 1 or earlier (including versions 9.0 PSP 4 Update 4 and earlier) as in some cases this firmware on

Page | 36 The Host Server – Microsoft Windows Configuration Guide


Previous changes

the Host can cause the DataCore Server to crash.

2 or 4 GB QLogic Fibre Channel Host Bus Adaptors


Do not use driver versions 9.0.1.12 or 9.0.2.11 as they may cause unexpected disconnections from the DataCore Server.

These were both previously documented in 'Known Issues - Third Party Hardware and Software'
http://datacore.custhelp.com/app/answers/detail/a_id/1277

Removed
Known Issues

When choosing a preferred Server setting for Windows Hosts with ALUA enabled, do not select 'All' setting
when using SANsymphony 10.0.4.x or greater
Choose an explicit DataCore Server (or leave as 'Auto Select') as in some cases when the 'ALL' setting is used,
failover may not work as expected when one of the DataCore Servers is stopped.

The root cause of this is now understood and so has been superseded by the known issues that were added in
March 2017:

Microsoft's multipath bus driver (MPIO.sys) may not failover if two or more paths from a Host to a Virtual Disk fail at the
same time, or if the same Virtual Disk for two or more paths is unavailable.

Even with the 'ALL' setting is used, as long as there is only one FE connection for a mirrored vDisk to each DataCore Server,
failover will work as expected.

March
Added
Known Issues
Microsoft's multipath bus driver (MPIO.sys) may not failover if two or more paths from a Host to a Virtual Disk fail
at the same time, or if the same Virtual Disk for two or more paths is unavailable.

February
Added
The Windows Host's Settings – Operating System Settings – Windows Updates
DataCore strongly recommend apply the 'December 2016 Security Monthly Quality Rollup for Windows 8.1 and
Windows Server 2012 R2' on all Windows 2012 R2 DataCore Servers.

Updated
DataCore MPIO for Windows – Compatibility list
All previously 'Supported' entries have been marked 'Supported (EOL)' for those users that are still using old
versions of Windows with this version of DataCore's MPIO to indicate what was compatible when the product was
officially End of Life.

Entries that were marked as 'not qualified' have now been changed to 'not supported' as DataCore will not be able
to accept self-qualification of an End-of-Life product. Otherwise all statements regarding support for end users
using End-of-Life and Not Supported combinations still apply.

January
Added
VSS compatibility lists

Page | 37 The Host Server – Microsoft Windows Configuration Guide


Previous changes
Includes Microsoft's VSS Provider and DataCore's VSS Provider (via the HIK).

The Windows Hosts Settings


Added notes for Windows 2016 Hosts regarding Operating System type.

Appendix C – Reclaiming Storage


Added note that Windows 2016 Hosts are included for SCSI UNMAP

Updated
Microsoft Windows compatibility and MPIO compatibility lists
The table originally used in the Microsoft Windows compatibility section was not needed as the separate MPIO
compatibility lists are more specific. This entire table has been removed.

2016

December
Initial Publication

Added
Microsoft Windows compatibility and MPIO compatibility lists
Added notes for Windows Server 2016

Updated
Appendix C - Reclaiming storage
Automatic and Manual reclamation
These two sections have been re-written with more detailed explanations and technical notes.

The most recent version of this document is available from here:


http://datacore.custhelp.com/app/answers/detail/a_id/838

Included from previous documentation

• The Host Server - Qualified Software Components


• SANsymphony Release Notes
Known Issues - For Windows hosts, do not use the "All" option when setting a "Preferred Server" in the host
details. Either specify a DataCore server, or use the "Auto select" setting.

• Technical Bulletins
# 11 - Recommended Disk Timeout Settings for Host Servers
# 13 - Microsoft Windows Cluster configuration guide

• Information previously documented in FAQs:


# 475 - DataCore MPIO not using the primary volume or preferred DataCore Server after start up
#1544 - Formatting a Virtual Disk with Windows 2012 Hosts may take longer than expected
#1586 - SANsymphony-V and Offloaded Data Transfer (ODX) with Windows 2008 and 2012 Hosts

Page | 38 The Host Server – Microsoft Windows Configuration Guide


COPYRIGHT

Copyright © 2018 by DataCore Software Corporation. All rights reserved.


DataCore, the DataCore logo and SANsymphony are trademarks of DataCore Software Corporation. Other DataCore product or service names
or logos referenced herein are trademarks of DataCore Software Corporation. All other products, services and company names mentioned
herein may be trademarks of their respective owners.

ALTHOUGH THE MATERIAL PRESENTED IN THIS DOCUMENT IS BELIEVED TO BE ACCURATE, IT IS PROVIDED “AS IS” AND USERS MUST TAKE ALL
RESPONSIBILITY FOR THE USE OR APPLICATION OF THE PRODUCTS DESCRIBED AND THE INFORMATION CONTAINED IN THIS DOCUMENT.
NEITHER DATACORE NOR ITS SUPPLIERS MAKE ANY EXPRESS OR IMPLIED REPRESENTATION, WARRANTY OR ENDORSEMENT REGARDING, AND
SHALL HAVE NO LIABILITY FOR, THE USE OR APPLICATION OF ANY DATACORE OR THIRD PARTY PRODUCTS OR THE OTHER INFORMATION
REFERRED TO IN THIS DOCUMENT. ALL SUCH WARRANTIES (INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, NON-
INFRINGEMENT, FITNESS FOR A PARTICULAR PURPOSE AND AGAINST HIDDEN DEFECTS) AND LIABILITY ARE HEREBY DISCLAIMED TO THE
FULLEST EXTENT PERMITTED BY LAW.

No part of this document may be copied, reproduced, translated or reduced to any electronic medium or machine-readable form without
the prior written consent of DataCore Software Corporation

Вам также может понравиться