Availibnilty

DIG64 Hot-Plug & Partitioning Flows (HPPF) Specification R1.
0
A publication of the Consortium of Developers Interface Guides for Intel Itanium Architecture-based Servers (DIG64) December 2007
Bull SAS Fujitsu Siemens Computers Hewlett-Packard Company Hitachi Limited Intel Corporation NEC Corporation Unisys Corporation
DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0
Notice: Implementations developed using the information provided in this specification may infringe the intellectual property rights of various parties including, but not limited to, the Promoters and parties involved in the development of this specification. An agreement to grant a limited patent license is available under the terms of a DIG64 Adopters Agreement, a copy of which may be obtained from http://www.dig64.org. THIS DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NON-INFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. The Promoters of this document disclaim all liability, including liability for infringement of any proprietary rights, relating to use of information in this specification. No license, express or implied, by estoppel or otherwise, to any intellectual property rights (including without limitation rights under any party's patents) is granted herein, except that a copyright license is hereby granted to you to copy and reproduce this specification for your internal use only. Copyright 2007 Bull SAS, Fujitsu Siemens Computers, Hewlett-Packard Company, Hitachi Limited, Intel Corporation, NEC Corporation, and Unisys Corporation (collectively referred to as the "DIG64 Promoters"). All rights reserved. *Other product and corporate names may be trademarks of other companies and are used only for explanation and to the owners benefit, without intent to infringe.
2 Copyright 2007 Bull SAS, Fujitsu Siemens Computers, Hewlett-Packard Company, Hitachi Limited, Intel Corporation, NEC Corporation, and Unisys Corporation (collectively referred to as the "DIG64 Promoters").
Table of Contents
HOT-PLUG & PARTITIONING FLOWS SPECIFICATION .............................................. 1 1
1.1 1.2
INTRODUCTION...................................................................................................... 6
Conventions and Terminology ............................................................................................6 Reference Documents..........................................................................................................8
2
2.1 2.1.1 2.1.2 2.1.3 2.1.4 2.2 2.2.1 2.2.2 2.2.3 2.2.4 2.2.5 2.3 2.3.1 2.3.2 2.3.3 2.3.4 2.4
PLATFORM HOT-PLUG FLOWS ......................................................................... 10

Memory Hot-Plug Flows.....................................................................................................10 Introduction ........................................................................................................................10 Memory Hot-Add Flow.......................................................................................................10 Memory Hot-Remove Flow................................................................................................13 Memory Hot-Plug - Required ACPI Primitives...................................................................16 Processor Board Hot-Plug.................................................................................................17 Introduction ........................................................................................................................17 Assumptions ......................................................................................................................17 Hot-Add Flow for Processor Boards (with Memory) ..........................................................17 Processor Board Hot-Remove Flow..................................................................................22 Processor Board Hot-Plug - Required ACPI Primitives.....................................................23 I/O Controller Hub Hot-Plug...............................................................................................24 Introduction ........................................................................................................................24 I/O Controller Hub Hot-Add Flow.......................................................................................24 I/O Controller Hub Hot-Remove Flow................................................................................27 I/O Controller Hub Hot-Plug - Required ACPI Primitives ..................................................28 ACPI to System Firmware Interface ..................................................................................29
3
3.1 3.1.1 3.1.2 3.1.3 3.2 3.2.1 3.2.2
PARTITIONING SCENARIOS THAT UTILIZE HOT-PLUG FLOWS .................... 30

Partition Unit Hot-Plug .......................................................................................................30 Introduction ........................................................................................................................30 ACPI Representation.........................................................................................................30 ACPI Sample Code ...........................................................................................................30 Dynamic Domain Partitioning............................................................................................32 Creating a new Domain Partition.......................................................................................32 Migrating resources between partitions.............................................................................33
This page intentionally left blank
1 Introduction
Servers based on the Intel Itanium Architecture, or IA-64, are generally designed for use in mission critical environments. Reliability, Availability and Serviceability (RAS) features such as the capability to add, remove, or replace processors, memory and I/O controller hubs without interruption of service are vital to achieving the highest levels of system availability ( 99.999% or better) required by these environments. In the area of domain partitioning, these hot-plug capabilities also provide the very mechanisms that enable dynamic allocation of processor, memory and I/O resources to partitions as well as between partitions. This document describes the architectural flows for hot-plug and partitioning on multiprocessor (MP) servers based on IA-64 (using Intel Itanium Architecture processors) and/or Intel64 (Intel Xeon processors with Extended Memory 64 Technology) architecture. These flows make use of standards-based platform interfaces such as ACPI and are meant to be implemented with OSs that support ACPI. Through this specification, DIG64 aims to promote and help accelerate industry adoption of RAS features across various MP platform designs and operating systems (OSs). As platform capabilities such as hot-plug subsystems and domain partitioning move from high-end servers into the mainstream of standard high volume servers, having common, architected flows for these features will help realize the benefits of interoperability and shorter development cycles while allowing for innovation and differentiation through the use of industry standards wherever possible.
1.1 Conventions and Terminology

The following terms are used in this specification: ACPI AP APIC Advanced Configuration and Power Interface Application Processor. For a given system partition, refers to all processors other than the System Bootstrap Processor. Advanced Programmable Interrupt Controller residing in the processor agents to generate and accept interrupt message from other processor and I/O agents. Some times it is referred as Local APIC. ACPI Source Language Basic Input/Output System. Also known as the system firmware of traditional PC platforms. Boot Strap Processor Refers to a constituent element of a system that performs a specific function. Refers to one of possibly multiple execution cores implemented on a single Intel processor. A core may contain one or more logical processors. Corrected Machine Check pertains to errors that originate from processors but are corrected in hardware. Corrected Platform Errors originate from non-processor sources such as the memory or I/O subsystem and are corrected in hardware. Corrected Platform Error Interrupt is used to signal a corrected platform error (CPE) to the OS. The interrupt vector for CPEI is set by the OS (i.e. in the vector
ASL BIOS BSP Component Core CMC CPE CPEI
field of an I/O SAPIC redirection table entry). CPEP Domain Partitioning Corrected Platform Error Polling refers to the sampling for the occurence of a corrected platform error as opposed to the use of an interrupt as in CPEI. The ability to divide a system into a number of smaller systems (or domains) each booting its own OS and operating independently of one another. It may also include the ability to merge multiple domains to form a single larger system (see Dynamic Partitioning). The type of domain partitioning that can be achieved without having to shut down the system or rebooting the OS of the partitions involved. This is in contrast to Static Partitioning. Extensible Firmware Interface Field Replaceable Unit, or a system component that can be physically added, removed and/or replaced in the field. Depending on the platform, an FRU can be hot-pluggable or not. The type of domain partitioning where each partition is completely isolated from software and hardware errors of the other partitions in the system. The physical addition of a FRU component to a running system. The hot-added component is recognized by the OS and added to the running system without shutting down. This capability typically requires special support from the OS. The generic term used to refer to Hot-Add and Hot-Remove operations. The ability to physically remove of an FRU component from a running system without having to stop the OS. This capability typically requires special support from the OS. The ability to physically replace an FRU from a running system without having to stop or shut down the OS. Hot-Replace actions are usually prompted by components that are defective or starting to show signs of defects. A Hot-Replace operation is basically a Hot-Remove followed by a Hot-Add. Hot-Replace capabilities can have restrictions based on platform and OS capabilities. For example, the memory device that contains physical address 0 may not be HotRemovable on some platforms. The ability to replace a component with a spare that has been readily prepared in the running system to serve as a replacement any time during normal system operation. In Hot-Swap situations, the OS resources on the failing component are migrated to the spare. Hot-Swap can be OS-transparent where the hardware is responsible for migrating the OS resources (e.g. DIMM sparing), or OS-assisted where the spare is visible to the OS and the platform migrates resources from the failing component to the spare in cooperation with the OS. Any of the processing units in a multiprocessing (MP) environment that the OS recognizes as a resource for executing a thread of code. Note that a single physical processor, or a single core within a multi-core processor, can have several logical processors. The logical (non-physical) removal or functional disabling of a component from a running system by an OS that has such capability. The offlined component can
Dynamic Partitioning EFI FRU
Hard Partitioning Hot-Add
Hot-Plug Hot-Remove
Hot-Replace
Hot-Swap
Logical Processor
Offlining
remain in the system as an inactive component (to be later removed during downtime) or reallocated to another partition within the system that is needing more resources. Onlining The logical (non-physical) addition or functional enabling of a component to a running system by an OS that has such capability. The onlined component can already be present in the system either as a spare ready to be activated, or as part of another partition that has offlined the component. (see Domain Partitioning) An Intel processor from the Itanium or Xeon processor family. A processor may contain one or more cores, where each core may contain one or more logical processors. An FRU board containing processors which connects to other boards in the system such as a motherboard or to a common backplane that interconnects all the boards in the system together. A processor board may also include memory controllers and DIMMs depending on the platform design. System bootstrap processor, or the monarch processor within a particular domain partition which hands off to the OS loader for OS boot on that partition. Domain partitioning that is configured only at boot time. This is in contrast to Dynamic Partitioning. Refers to a set of compute and I/O resources that can boot and run an OS as a single entity. Depending on context, a system can refer to a particular domain partition, or an entire platform that supports domain partitioning.
Paritioning Processor
Processor Board
SBSP Static Partitioning System
1.2 Reference Documents

Itanium Processor Family System Abstraction Layer (SAL) Specification (http://www.intel.com/design/Itanium/Downloads/245359.htm) Intel Itanium Architecture Software Developer's Manuals (http://www.intel.com/design/Itanium/manuals/iiasdmanual.htm) ACPI 3.0 Specification (http://acpi.info) PCI Firmware Specification, Revision 3.0 (http://www.pci-sig.com) DIG64 Corrected Platform Error Polling (CPEP) Interface Specification 1.0 (http://www.dig64.org) DIG64 Release Specification 2.2 (http://www.dig64.org)
2 Platform Hot-Plug Flows

This chapter describes the flows for the hot-addition, hot-removal, and hot-replacement of processors, memory and I/O controllers on platforms that provide hot-plug capabilities for these resources. These flows represent the typical scenarios where hot-pluggable compute resources can contribute to a platforms RAS. Note however that other platform elements that may be hot-pluggable such as power supplies and cooling fans, or I/O devices based on PCI Express or USB, are not covered here. This chapter describes the proposed hot-plug flows for a specified set of RAS features. The flows are built on top of standard platform interfaces such as ACPI and hence are widely applicable across OS implementation. The required ACPI primitives that OS needs to support are called out in each flow.
2.1 Memory Hot-Plug Flows

2.1.1 Introduction
Memory hot-plug allows for the modification of memory capacity of a running system. The granularity of memory hot-plug on a given platform is implementation-specific. The following assumptions are made regarding any platform that supports memory hot-plug: The systems memory interleaving configuration allows hot-plug memory in reasonable scales of granularity and operational complexity. Note that as interleaving of memory gets finer, memory hot-plug may be too difficult or even impractical to implement. Not every memory in the system may be hot-removable. There may be platform architectureand/or OS-specific requirements that preclude removal of certain memory ranges. The platform is capable of generating the appropriate ACPI events to signal the relevant hotplug actions.
2.1.2 Memory Hot-Add Flow

Figure 1 illustrates the flow for the hot-addition of memory. It shows the progression from the point at which a user physically inserts new memory into the system until the OS is able to use the new memory. Note the depiction of the ACPI Flow as a firmware-independent sequence although ACPI is normally implemented as part of system firmware.
Figure 1. Memory Hot-Add Flow
The following describes the steps of the memory hot-add flow in more detail:
2.1.2.1 Memory Hot-Add Hardware Flow

1) The user inserts the memory FRU into the system. 2) The user initiates a hot-add memory request via the mechanism provided by the system, which could be a button, a command from a management console or application. 3) The system powers up the hot-added memory device and triggers flow of system firmware (or BIOS on legacy implementations) leading to steps in section 2.1.2.2
2.1.2.2 Memory Hot-Add System Firmware Flow

1) System firmware verifies that the hot-added memory is consistent, for example, ensuring that there are no faulty DIMMs, mixed DIMM types, etc 2) System firmware initializes the hot-added memory 3) Initiate memory hot-add request to OS through ACPI (triggered using an ACPI GPE event) leading to the OS flow sequence below.
2.1.2.3 Memory Hot-Add ACPI Flow

This stage of the flow involves ACPI initialization for supporting memory hot-add. Traditionally, memory was only described as a resource in ACPI. Starting with ACPI 2.0, PnP ID PNP0C80 was defined to represent memory devices where the corresponding _CRS object describes the physical address space that the memory device decodes. This allowed for hot-plug of memory by sending the appropriate notifications to the PNP0C80 device. Due to memory interleaving, the physical address space decoded by the memory device may not be contiguous. If the physical address space decoded by the device can be reasonably described using _CRS, it is acceptable to describe the memory device using PNP0C80. Note that describing _CRS memory resource macros may be impractical in certain cases. For example, describing two memory devices of 2GB each that are interleaved on 1MB boundaries would have each _CRS object containing 2,048 memory resource macros.
2.1.2.4 Memory Hot-Add OS Flow

The OS Flow proceeds as follows (note the use of ACPI objects and mechanisms): 1) On receiving the System Control Interrupt (SCI), the OS invokes the corresponding GPE handler 2) The OS generates a Notify with Notify Code 1 to the corresponding ACPI device (either PNP0C80 or a Module device depending on the implementation) 3) If the OS is able to use the hot-added memory, it calls _OST to indicate success and evaluates the _CRS object for the memory device to identify the physical address space now supported with memory. 4) If the OS is not able to use the new memory, the OS calls _OST to indicate failure, after which the platform can allow the memory FRU to be removed without further OS interaction.
2.1.2.5 ACPI Sample Code

Memory Hot-Plug ASL: \_SB { Device (MD1) { //Memory Device Name(_HID, EISAID(PNP0C80)) Name (_CRS, ResourceTemplate) (0x4000_0000 to 0x7FFF_FFFF) // 1GB range [230..230+1GB-1]
Method (_STA, 0) { // If memory not present --> Return(0x00) // Else if memory is disabled --> Return(0x0D) // Else --> Return(0x0F) // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
} // MD1 Device (MD2) { //Memory Device Name(_HID, EISAID(PNP0C80)) Name (_CRS, ResourceTemplate) (0x8000_0000 to 0xBFFF_FFFF) // 1GB range [231..231+1GB-1]
Method (_STA, 0) { // If memory not present --> Return(0x00) // Else if memory is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
} // MD2 } // _SB scope Scope (\_GPE) { Method _Lxx { If (MD1 add) Notify(MD1,1) If (MD2 add) Notify(MD2,1) If (MD1 remove) Notify(MD1,3) If (MD2 remove) Notify(MD2,3) } // _Lxx } // _GPE
2.1.3 Memory Hot-Remove Flow

The memory hot-remove flow is shown in Figure 2. Note it is mainly an OS flow; as mentioned earlier, the affected address ranges are assumed to be controlled by the OS.
IMPLEMENTATION NOTE: Hot-remove capable memory present in the system at boot time must be described as hot-removable using the System Resource Affinity Table (SRAT) of ACPI.
Hardware Flow
System Firmware and ACPI Flow
OS Flow
User initiates memory hotremove request (via button press, management console or application)
Trigger ACPI event
Issue ACPI Notify to OS to indicate memory hot-remove request

Notify to OS
Issue hot-remove Notify to corresponding ACPI Module Device
OS attempts to offline the specified memory
OS flushes all references to specified memory from all caches
Was OS successful in offlining memory?
Call _EJ0
Call _OST with error code
Figure 2. Memory Hot-Remove Flow
The following describes the steps of the memory hot-remove flow in more detail. Note that the sample code template for memory hot-add provided in 2.1.1.5 can also be used for memory hotremove.
2.1.3.1 Memory Hot-Remove Hardware Flow

1) The user initiates hot-remove request (via button press, management console or application) 2) System indicates hot-remove request to OS through ACPI event leading to sequence in section 2.1.3.2
2.1.3.2 Memory Hot-Remove ACPI and OS Flows

1) On receiving the ACPI interrupt (SCI), the corresponding GPE handler is invoked 2) OS issues a Notify with Notify Code 3 to the corresponding ACPI memory device (which could be PNP0C80 or Module Device, depending on the implementation) 3) OS determines if the memory range can be offlined.
Some ranges of physical memory may be treated as special by some OSs such that they cannot be offlined and therefore not hot-removable. The memory range targeted for hot-removal may contain pinned pages that prevent successful offlining
4) If the OS determines the memory can be offlined, the OS attempts to offline the specified memory from the running system. The OS migrates the contents from the specified memory range, releases physical memory pages and removes page table entries that belong to the specified range. 5) OS flushes all the processor caches to purge all cache lines of references to the target memory range. IMPLEMENTATION NOTE on Itanium Architecture: The following describes a scheme for flushing processor caches on Itanium Architecture platforms. First, a sequence for halting speculative references and prefetches to the target range and for returning cache lines to the memory being removed is described in The Intel Itanium Architecture Software Developers Manual, Volume 2, Revision 2.2, Volume 2, Section 4.4.11.3, here referred to as SDM Sec 4.4.11.3. The OS basically uses this sequence under the following requirements: 1. The OS must execute the SDM Sec 4.4.11.3 sequence on all processors owned by the OS at the time of the hot-removal operation. 2. The OS must wake and execute the SDM Sec 4.4.11.3 sequence on all processors on which the OS had called PAL_HALT. The SAL is not responsible for processors placed in the HALT state by the OS 3. In step 9 of the SDM Sec 4.4.11.3 sequence, the OS must call SAL_CACHE_FLUSH from every processor active in the OS to flush all platform caches. However, there may be processors that are not visible to the OS and are visible only to the SAL at the time of the hot-remove request (e.g. some processors may not be visible to the OS due to license restrictions, or returned by the OS to SAL BR0 loop). In this case, the SAL needs to ensure these processors are performing prefetches for only code and data owned by firmware. This is accomplished by executing the following sequence on every processor visible only to the SAL: 1. Call PAL_PTCE_INFO to get information needed to purge translation caches, then use the parameters returned to execute the loop describing the ptc.e (purge translation cache entry) instruction in The Intel Itanium Architecture Software Developers Manual, Revision 2.2, Volume 3, Section 2.2. 2. Call PAL_PREFETCH_VISIBILITY with trans_type=1 3. Call PAL_CACHE_FLUSH with the inv parameter set to 1. 4. Call PAL_MC_DRAIN The execution of the above sequence by the SAL for processors visible only to the SAL must meet the following requirements:
1. SAL firmware must execute this sequence when the OS returns the processor to system firmware via the BR0 return address. 2. SAL firmware must execute this sequence whenever a processor that has completed Selftest is placed in the BR0 loop to wait to be claimed by the OS (as in after performing memory test during initialization). 3. This sequence must be executed from the SAL memory area where the SAL BR0 loop is located without making references to any other memory area except to the PAL procedure calls used in the sequence. 4. No references to any OS-owned memory area may be made during or after execution of the sequence.
6) If OS successfully offlines the memory from running system, it calls _EJ0 of the corresponding module device to indicate successful offlining of the memory by OS. Since _EJ0 does not return status, OS will evaluate _STA method to verify the success of the ejection operation. 7) If _EJ0 is called, ACPI code may invoke system firmware to continue to hot-remove operation (e.g. power off the memory FRU device). 8) If OS is unable to offline the memory, then it calls _OST to indicate failure to ACPI code.
2.1.4 Memory Hot-Plug - Required ACPI Primitives

Memory hot-plug (hot-add and hot-remove) flows require that the OS support a number of ACPI primitives. TheACPI primitives that are required for memory hot-plug are. ACPI Memory Device (Device ID PNP0C80) _OSI (object for representing Operating System Interfaces) _OST (object for status indication of device insertion and removal) _STA (object for status of device enabled/disabled, insertion/removal)
Note that this is not an exhaustive list of every ACPI object that needs to be implemented. Only the relevant ACPI primitives that were either introduced in ACPI 3.0 or that have not been universally supported in ACPI 2.0 compliant OSs are specifically mentioned here.
2.2 Processor Board Hot-Plug

2.2.1 Introduction
In this document, a processor board is by definition (see Section 1.1) a hot-pluggable module on a platform designed for such capability. A processor board may consist of one or more processors and may or may not include memory controllers populated with memory, depending on the platform design. In the flows for described below, it is assumed that a processor board has at least one memory controller with attached memory.
2.2.2 Assumptions
The platform does not have power domain dependencies between separate processor boards. Given that a processor board is assumed to include memory, if the platform allows memory interleaving across processor boards, the same assumptions pertaining to memory interleaving described in the Memory Hot-Plug section apply here as well. If CPEI (Corrected Platform Error Interrupt) is used, the OS must comprehend the CPEI Processor Override Flag in the Platform Interrupt Source Structure (defined in ACPI 3.0) to support hot-removal of a processor board that is targeted for a CPEI. Note that the ACPI 2.0 definition of CPEI implies that the CPEI handler can only be run on a specific processor. ACPI 3.0 relaxes this restriction by making the processor IDs in the Platform Interrupt Source Structure provide a hint for a target processor (see ACPI 3.0 Section 5.2.11.14). If CPEP (Corrected Platform Error Polling) is used, the OS needs to comprehend _PPE (Poll for Corrected Platform Error) method handling during device notification as outlined in the DIG64 CPEP Interface Specification (see References). The OS must comprehend Processor ID extension using _UID (ACPI 3.0) if it is supporting processor hot-plug on a system with > 256 processors Any processor board can be hot-removed. Note that there may be OS limitations on hotremoval of the System Bootstrap Processor (SBSP), and/or boot memory that happens to reside on the processor board. If hot-remove of a SBSP is supported, during the hot-removal of the SBSP, one of the APs (Application Processors) must contain the return value to SAL.
2.2.3 Hot-Add Flow for Processor Boards (with Memory)

The flow for hot-add of a processor board (with memory) is shown in Figure 3. Note the similarity with the memory hot-add flow.
Figure 3. Processor Board Hot-Add Flow
The following describes the steps involved in Figure 3 in more detail.
2.2.3.1 Processor Board Hot-Add - Hardware Flow

1) The user inserts the new processor board into the system 2) The user initiates a processor board hot-add request by some platform-dependent mechanism such as a mechanical button or by issuing a command on a management console or application. 3) The platform powers up the processor board and starts the firmware flow to initialize it.
2.2.3.2 Processor Board Hot-Add System Firmware Flow

1) System firmware updates the system topology to comprehend new processor board 2) System firmware triggers an ACPI GPE event leading to the OS flow sequence in section 2.3.3.4
2.2.3.3 Processor Board Hot-Add ACPI Flow

This stage involves ACPI initialization for supporting processor board hot-add. The ACPI namespace provided by system firmware at boot time describe the processor boards as ACPI Module devices. Because legacy OSs do not comprehend Module devices, the ACPI code must identify the OS and its capabilities (using _OSI) and only expose Module device representation if the OS supports Module devices.
2.2.3.4 Processor Board Hot-add OS Flow

1) On receiving the SCI, the OS invokes the corresponding GPE handler 2) The OS generates a Notify with Notify Code 1 to the corresponding ACPI Module device for the processor board 3) The OS enumerates the processor board and adds processor(s) and memory to the running system OS evaluates _MAT to identify Local APIC/SAPIC and local NMI entries OS evaluates the _CRS object for memory devices to determine the system address range now supported by the new processor boards memory OS evaluates the processor performance control objects OS evaluates the proximity using the _PXM method (or _SLI objects if SLIT is present) to determine the proximity domain of the newly introduced processors and memory (they should be in the same proximity domain) OS adds the new processors to the power management domain of the OS 4) If OS is able to support the new processor board, it calls _OST to indicate success; otherwise, OS calls _OST to indicate failure. The OS is expected to call _OST to indicate failure of hotadd even if the corresponding ACPI object implements the _EJ0 method. If OS calls _OST to indicate failure, the platform can allow the processor board FRU to be removed without further OS interaction.

ASL to detect OS support for Module Devices \_SB { Method (_INI) { // detect if OS supports Module Device If (CondRefOf (_OSI, Local0)) { If (\_OSI ("Module Device")) { // OS supports Module Device // Represent processor board as a Module Device } Else { // no Module Devices // Represent cores within the processor board // as Processor Devices. } } Else { // no _OSI support in OS // Represent cores within the processor board // as Processor Devices. } } // _INI
} // _SB scope ASL for OS that supports Processor Board Hot-Add and Hot-Remove Scope (\_SB) { Device (S1) { // Processor Board1
Name(_HID, ACPI0004) // Module Device IMPLEMENTATION NOTE: DIG64 recommends using ACPI Module Device objects to describe nodes in a NUMA system. Refer to DIG64 R2.2, section 5.5.2.4.
Method (_STA, 0) { // If Module is not present --> Return(0x00) // Else if Module is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to Module }
Method (_PXM) {} Processor( CPU1, 0, 0, 0) { Method (_MAT) {} } // CPU1 Processor( CPU4, 0, 0, 0) { Method (_MAT) {} } // CPU4 Device (MD1) {//Memory Device Name(_HID, EISAID(PNP0C80)) Name (_CRS, ResourceTemplate) (0x4000_0000 to 0x7FFF_FFFF) // 1GB range [230..230+1GB-1]
Method (_STA, 0) { // If memory device not present --> Return(0x00) // Else if memory device is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
} // MD1 Device (MD2) {//Memory Device Name(_HID, EISAID(PNP0C80)) Name (_CRS, ResourceTemplate) (0x8000_0000 to 0xBFFF_FFFF) // 1GB range [231..231+1GB-1)
Method (_STA, 0) {
// If memory device not present --> Return(0x00) // Else if memory device is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
} // MD2 } // S1 } // _SB Scope (\_GPE) { Method _Lxx { If (S1 hot-add request) Notify (S1, 1) If (S1 hot-remove request) Notify (S1, 3) } // _Lxx } // _GPE
2.2.4 Processor Board Hot-Remove Flow

Figure 4 shows the processor board hot-remove flow. Note the similarity with memory hot-remove.
Hardware Flow
System Firmware and ACPI Flow
OS Flow
User initiates processorboard hot- remove request (via button press, management console or application)
Trigger ACPI event
Issue ACPI Notify to OS to indicate processor board hotremove request

Notify to OS
Issue hot-remove Notify to corresponding ACPI Module device
OS attempts to offline the processors of the processor board
OS attempts to offline the memory of the processor board
OS flushes all references to the targeted memory from all caches
Was OS successful in offlining the processor(s) and memory?
Call _EJ0
Figure 4. Processor Board Hot-Remove Flow
2.2.4.1 Processor Board Hot-Remove Hardware Flow

1) The user initiates processor board hot-remove request to OS 2) The platform initiates an ACPI hot-remove event which involves generating an SCI
2.2.4.2 Processor Board Hot-Remove ACPI and OS Flow

1) On receiving the SCI, the OS invokes the corresponding GPE handler.
2) The OS generates a Notify with Notify Code 3 (eject request) to the ACPI device corresponding to the processor board 3) OS attempts to offline the processors before offlining memory corresponding to the processor board. The process by which the OS offlines processors is specific to the OS. However, platform firmware can assume that on completion of offlining of processors by the OS, the corresponding caches of the processors have been flushed by the OS. The details of this process are described in the section on memory hot-remove. 4) If the OS successfully offlines the resources (processors and memory) on the processor board from the running system, it then calls _EJ0 of the processor board device. Since _EJ0 does not return status, the OS evaluates _STA method to verify the success of the ejection operation.
IMPLEMENTATION NOTE for Itanium Architecture: The OS is required to return the offlined processors to a SAL rendezvous loop, prior to calling _EJ0 to indicate successful offlining of the processor board. This note assumes offlining of non-BSP processors only.
5) If _EJ0 is called, ACPI code may invoke system firmware to continue the hot-remove operation, for example. cutting off power to the processor board.
IMPLEMENTATION NOTE for Itanium Architecture: Prior to removing power, SAL needs to ensure that all the processors in the SAL boot rendezvous routine for the outgoing processor board are placed in PAL_SHUTDOWN state. This ensures that all the processors enter a low power state before the power is removed from the processor board.
6) If the OS is unable to offline the processor board, it calls _OST to indicate failure of the hotremove operation.

The sample code is the Memory Hot-Plug section (2.1.1.5) provides a template for dealing with the memory portion of the processor board.
2.2.5 Processor Board Hot-Plug - Required ACPI Primitives

Processor board hot-plug (hot-add and hot-remove) flows described above require OS support the following ACPI primitives. ACPI Module Device (Device ID ACPI0004)
_OSI (Object for representing Operating System Interfaces) _OST (Object for status indication for device insertion and removal) _MAT (Object for describing a Multiple APIC Device) _PXM (Object for describing proximity domains of devices) _SLI (System Locality Information) , if SLIT (System Locality Distance Information Table) is supported Comprehension of the CPEI Override Flag _STA (Object for status of device enabled/disabled, insertion/removal)
Note that this is not an exhaustive list of every ACPI object that needs to be implemented. Only the relevant ACPI primitives that were either introduced in ACPI 3.0 or that have not been universally supported in ACPI 2.0 compliant OSs are specifically mentioned here.
2.3 I/O Controller Hub Hot-Plug

2.3.1 Introduction
An I/O Controller Hub is a host bridge device that has hot-plug capabilities within the context of the bus or interconnect technology implemented on the platform. The hot-pluggable entity consists of an I/O subsystem rooted in a ACPI host bridge device (either PNP0A03 or PNP0A08). The I/O hierarchy may include a number of I/O devices directly attached to the host bridge or to other bridges that may be present further below in the hierarchy. Hot-plug of an I/O Controller Hub includes the capability for hot-addition and hot-removal of the controller including the I/O devices behind it. The following assumptions are made with regards to I/O Controller Hub hot-plug flows to be described: Hot-addition/removal of subtractive decode resources is not supported. The OS supports ACPI host bridge hot-plug. The OS supports I/O APIC hot-plug, since the I/O hierarchy may include several I/O APICs (IOSAPIC or IOxAPIC).
2.3.2 I/O Controller Hub Hot-Add Flow

The I/O Controller Hub hot-add flow is shown in Figure 5 and described in more detail below.
Figure 5. I/O Controller Hub Hot-Add Flow
2.3.2.1 I/O Controller Hub Hot-Add Hardware Flow

1) The user inserts I/O controller hub FRU into the system. 2) The user initiates hot-add of I/O controller hub (via button press, management console or application) 3) The system powers on the inserted I/O controller hub 4) The system issues a hot-add request to system firmware leading to the firmware flow below.
2.3.2.2 I/O Controller Hub Hot-Add System Firmware Flow

1) The system firmware initializes the I/O controller hub and the needed I/O devices behind it. Refer to the PCI Firmware Specification Rev 3.0 (section 3.5, Device State at Firmware/OS Handoff) for device state initialization requirements (see References). 2) The system firmware initiates an I/O controller hub hot-add request to the OS through ACPI (by generating GPE Event) leading to the OS flow in section 2.3.2.4
2.3.2.3 I/O Controller Hub Hot-Add ACPI Flow

The ACPI namespace at boot time must include representation for hot-pluggable I/O controller hubs in the system. Each I/O controller hub is represented as a ACPI host bridge. The MMIO, I/O and PCI Bus resources for each host bridge is pre-determined at system startup.
If the I/O Controller Hub contains I/O APIC devices, the I/O APIC device must be described in the ACPI namespace in order to provide the global system address base of the I/O APIC (IOSAPIC or IOxAPIC).
2.3.2.4 I/O Controller Hub Hot-Add OS Flow

1) On receiving the SCI, the OS invokes the corresponding GPE handler. 2) The OS generates a Notify with Notify Code 1 to the corresponding ACPI host bridge device 3) If the OS is able to support the hot-add I/O controller hub request, it calls _OST to indicate success; otherwise, the OS calls _OST to indicate failure. If the OS calls _OST to indicate failure, the platform can allow removal of the I/O controller hub FRU without further OS interaction. 4) If the OS is able to support the hot-add of the I/O controller hub, the OS discovers and configures the hierarchy of I/O devices below the hot-added I/O Controller Hub.

ASL for I/O Controller Hub Hot-Plug \_SB { ... Device (IOH1) { // host bridge representation for IOH1 Name (_HID, EISAID(PNP0A08)) Name (_CID, EISAID(PNP0A03))
Method (_STA, 0) { // If IOH not present --> Return(0x00) // Else if IOH is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
... } Device (IOH2) { // host bridge representation for IOH2 Name (_HID, EISAID(PNP0A08)) Name (_CID, EISAID(PNP0A03)) Name (_ADR, 0)
Method (_STA, 0) { // If IOH not present --> Return(0x00) // Else if IOH is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
Method (_CRS) {} Name (_SEG, 0) // Name (_BBN, ) // Method (_CBA) {}
// resources for host bridge indicates PCI Segment for the IOH2 indicates boot bus number for IOH2 // return MMCFG base address for // host bridge
26
Copyright 2007 Bull SAS, Fujitsu Siemens Computers, Hewlett-Packard Company, Hitachi Limited, Intel Corporation, NEC Corporation, and Unisys Corporation (collectively referred to as the "DIG64 Promoters").
Method (_OSC, 4) { } // enable OS to indicate //capabilities related to PCI-E Device (PCI1) { // IOAPIC PCI Device Name (_ADR, ) } ... } } // _SB scope Scope (\_GPE) { Method _Lxx { if (IOH2 hot-add) Notify(IOH2, 1) if (IOH2 hot-remove) Notify(IOH2, 3) } // _Lxx } // _GPE
2.3.3 I/O Controller Hub Hot-Remove Flow

I/O Controller Hub hot-remove flow is shown below. The individual elements of the flow are explained further in subsequent sections.
Hardware Flow
Platform Firmware Flow
OS Flow
User initiates hot- remove request
Issue ACPI Notify to OS to indicate I/O controller hub hot-remove request

Notify to OS
Generate hot-remove Notify to corresponding ACPI Host Bridge device
Trigger ACPI event
OS attempts to offline the I/O controller hub and the hierarchy of devices under it
Was OS successful in offlining the I/O controller hub?
Call _EJ0
Figure 6. I/O Controller Hub Hot-Remove Flow
The flow is described in more detail as follows.
2.3.3.1 I/O Controller Hub Hot-Remove Hardware Flow

1) The user initiates hot-remove request 2) Platform initiates ACPI event to indicate hot-remove request leading to the following steps.
2.3.3.2 I/O Controller Hub Hot-Remove ACPI and OS Flow

1) On receiving the SCI, the OS invokes the corresponding GPE handler. 2) The OS generates a Notify with Notify Code 3 to the corresponding ACPI host bridge device 3) The OS attempts to offline the hierarchy of I/O devices behind the I/O controller hub 4) If the OS successfully offlines the I/O controller hub and the devices behind it, the OS calls _EJ0 of the corresponding ACPI host bridge device to indicate success 5) If _EJ0 is called, ACPI code invokes system firmware to continue the hot-remove operation, for example, powering off the I/O controller hub. 6) If the OS is unable to offline the I/O controller hub and its resources, then it calls _OST to indicate failure to the ACPI code.

See the sample code in the section on I/O controller hub hot-add.
2.3.4 I/O Controller Hub Hot-Plug - Required ACPI Primitives

The I/O controller hub hot-plug flows described here requires that the OS support the following ACPI primitives: _OSC (Object for representing OS Capabilities) _SEG (Representation for a PCI Segment Number) if the platform has multiple segments _CBA (Representation for the Configuration Base Register for a PCI Express host bridge). For DIG64-compliant Itanium architecture systems, the OS and higher layers of software must use the SAL_PCI_CONFIG_READ and SAL_PCI_CONFIG_WRITE procedures to access the PCI configuration space during runtime. DIG64-compliant systems do not support the _CBA method or the ACPI MCFG table (memory mapped configuration space base address description see PCI Firmware Specification under References) . Comprehension of ACPI Device ID PNP0A08 if PCI Express is supported _OST (Object for status indication of device insertion and removal). _STA (Object for status of device enabled/disabled, insertion/removal)
Note that this is not an exhaustive list of every ACPI object that needs to be implemented. Only the relevant ACPI primitives that were either introduced in ACPI3.0 or that have not been universally supported in ACPI 2.0 compliant OSs are mentioned here.
2.4
ACPI to System Firmware Interface
When implementing the flows specified in this document, it may be necessary for the ACPI code to invoke native system firmware to assist in completing the hot-plug flow. The mechanism to invoke native system firmware from ACPI code is implementation specific. IMPLEMENTATION NOTE: Using firmware-triggered platform management interrupts (PMI on Itanium Architecture, or SMI on Intel64 architecture), or defining special ACPI Operation Regions are possible approaches for implementation.
3 Partitioning Scenarios That Utilize Hot-Plug Flows

This chapter describes the partitioning scenarios that could be implemented using the hot-plug flows described in the previous sections.
3.1 Partition Unit Hot-Plug

3.1.1 Introduction
A partition unit (or simply partition) consists of one or more processors, memory and I/O controller hubs. Partition unit hot-plug can be built on top of the ingredient hot-plug flows described in previous sections. Any key differences from the ingredient flows are described in the following sections
3.1.2 ACPI Representation

ACPI namespace provided by system firmware at boot time represents partition units using ACPI Module Device. Because legacy OSs do not comprehend ACPI Module devices, the ACPI code must identify the OS and detect its capabilities (using _OSI) and only expose Module device representations if the OS supports Module devices.
3.1.3 ACPI Sample Code

ASL to detect OS support for Module Devices \_SB { Method (_INI) { // detect if OS supports module device If (CondRefOf (_OSI, Local0)) { If (\_OSI ("Module Device")) { // OS supports Module device } Else { // no module device } } Else { // no _OSI support in OS } } // _INI } // _SB scope
ASL for OS that supports Partition Unit Hot-Add and Hot-Remove Scope (\_SB) { Device (S1) { // Partition Unit1 Name(_HID, ACPI0004) // Module Device IMPLEMENTATION NOTE: DIG64 recommends using ACPI Module Device objects to describe nodes in a NUMA system. Refer to DIG64 R2.2, section 5.5.2.4.
Method (_STA, 0) { // If Module not present --> Return(0x00) // Else if Module is disabled --> Return(0x0D) // Else --> Return(0x0F) } // _STA Method(_EJ0, 1) { // Offline or remove all power to device }
Method (_PXM) {} Processor( CPU1, 0, 0, 0) { Method (_MAT) {} ... } // CPU1 Processor( CPU4, 0, 0, 0) { Method (_MAT) {} ... } // CPU4 Device (MD1) {//Memory Device Name(_HID, EISAID(PNP0C80)) Name (_CRS, ResourceTemplate) (0x4000_0000 to 0x7FFF_FFFF) // 1GB range [230..230+1GB-1]
Method (_STA, 0) {...} Method(_EJ0, 1) {...}
} // MD1 Device (MD2) {//Memory Device Name(_HID, EISAID(PNP0C80)) Name (_CRS, ResourceTemplate) (0x8000_0000 to 0xBFFF_FFFF) // 1GB range [231..231+1GB-1]
Method (_STA, 0) {...} Method(_EJ0, 1) {...}
} // MD2 Device (IOH1) { // host bridge representation for IOH1 Name (_HID, EISAID(PNP0A08))
Name (_CID, EISAID(PNP0A03)) ... }

Method (_STA, 0) {...} Method (_EJ0, 1) {...}
} // S1 Device (IOH0) { // host bridge representation of legacy IOH Name (_HID, EISAID(PNP0A08)) Name (_CID, EISAID(PNP0A03)) ... } } // _SB Scope (\_GPE) { Method _Lxx { If (S1 hot-add request) Notify (S1, 1) If (S1 hot-remove request) Notify (S1, 3) } // _Lxx } // _GPE
3.2 Dynamic Domain Partitioning

Domain partitioning involves creating new domains by reassigning resources currently associated with other domains. Domain partitioning involves one or more Onlining/Offlining operations followed by platform specific actions to establish domain perimeter. As a result, OS visible part of domain partitioning is already comprehended by the Onlining/Offlining flows in previous sections.
3.2.1 Creating a new Domain Partition

The following describes how a new partition can be created dynamically using the hot-plug capabilities described earlier. Consider a system with the following hot-plug capabilities: 4 processor boards each consisting of one processor and memory: PB0-PB3 where PB0 containing boot memory. 4 I/O boards each consisting of an I/O controller hub and some I/O devices behind it: IOH0IOH3 where IOH0 has legacy devices behind it
First, resources of an existing partition are offlined. For example, the following hot-plug FRUs can be offlined: PB2, PB3, IOH2, IOH3. Note hot-removal or offlining of PB0 or IOH0 may not be allowed due to OS and/or platform restrictions.
Next, the domain perimeter of the existing partition is reconfigured to remove the offlined components from the current domain. The mechanism for reconfiguring domain perimeters is implementation specific. The offlined components can now be used to construct a new domain partition using the onlining mechanisms supported on the platform.
3.2.2 Migrating resources between partitions

Consider the system is currently divided into two partitions each with the same hot-plug capable processor boards and I/O boards as in the previous example as follows: Partition1 4 processor boards: PB0, PB1, PB2, PB3 where PB0 contains boot memory 4 I/O boards: consisting of IOH (I/O Hub) and I/O devices behind the IOH (IOH0, IOH1, IOH2, -IOH3 where IOH0 has is the legacy I/O devices behind it
Partition2 4 processor boards: PB4, PB5, PB6, PB7 where PB4 contains boot memory 4 I/O boards: IOH4, IOH5, IOH6, IOH7 where IOH4 has legacy I/O devices behind it
The sequence of actions for migrating a processor board from Partition2 to Partition1 is as follows: 1. Perform a processor board offline on PB7 in Partition2 2. Reprogram the system domain perimeter to move PB7 from Partition2 to Partition1 (this step is implementation specific) 3. Online PB7 in Partition1.

Availibnilty

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Availibnilty

Загружено:

Авторское право:

Доступные форматы

DIG64 Hot-Plug & Partitioning Flows (HPPF) Specification R1.

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

PLATFORM HOT-PLUG FLOWS ......................................................................... 10

PARTITIONING SCENARIOS THAT UTILIZE HOT-PLUG FLOWS .................... 30

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

This page intentionally left blank

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

1.1 Conventions and Terminology

ASL BIOS BSP Component Core CMC CPE CPEI

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

Dynamic Partitioning EFI FRU

Hard Partitioning Hot-Add

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

SBSP Static Partitioning System

1.2 Reference Documents

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2 Platform Hot-Plug Flows

2.1 Memory Hot-Plug Flows

2.1.2 Memory Hot-Add Flow

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

Figure 1. Memory Hot-Add Flow

2.1.2.1 Memory Hot-Add Hardware Flow

2.1.2.2 Memory Hot-Add System Firmware Flow

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2.1.2.3 Memory Hot-Add ACPI Flow

2.1.2.4 Memory Hot-Add OS Flow

2.1.2.5 ACPI Sample Code

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2.1.3 Memory Hot-Remove Flow

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

System Firmware and ACPI Flow

Issue ACPI Notify to OS to indicate memory hot-remove request

Issue hot-remove Notify to corresponding ACPI Module Device

OS attempts to offline the specified memory

OS flushes all references to specified memory from all caches

Was OS successful in offlining memory?

Call _OST with error code

Figure 2. Memory Hot-Remove Flow

2.1.3.1 Memory Hot-Remove Hardware Flow

2.1.3.2 Memory Hot-Remove ACPI and OS Flows

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2.1.4 Memory Hot-Plug - Required ACPI Primitives

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2.2 Processor Board Hot-Plug

2.2.3 Hot-Add Flow for Processor Boards (with Memory)

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

Figure 3. Processor Board Hot-Add Flow

The following describes the steps involved in Figure 3 in more detail.

2.2.3.1 Processor Board Hot-Add - Hardware Flow

2.2.3.2 Processor Board Hot-Add System Firmware Flow

2.2.3.3 Processor Board Hot-Add ACPI Flow

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2.2.3.4 Processor Board Hot-add OS Flow

2.2.3.5 ACPI Sample Code

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

DIG64 Hot-Plug & Partitioning Flows Specification Release 1.0

2.2.4 Processor Board Hot-Remove Flow

System Firmware and ACPI Flow