Intel® Xeon® Processor E78800/4800/2800 Product Families Datasheet Volume 2 of 2 April 2011 Document Number: 325120-001
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.
Contents 1 Introduction .............................................................................................................. 7 1.1 Intel® Xeon® Processor E7-8800/4800/2800 Product Families Features...................... 7 1.2 Terminology and Conventions ............................................................................... 9 1.2.1 Abbreviations .......................................................................................... 9 1.3 Notational Conventions .........................
9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.1.1 Thermal Monitoring - 2 (TM2)...................................................................39 9.1.2 Thermal Monitoring - 1 (TM1) and T-state ..................................................39 9.1.3 THERMTRIP# .........................................................................................40 9.1.4 PROCHOT# ............................................................................................40 9.1.5 FORCEPR#..............................................
Revision History Document Number Revision Number 325120 001 Description • Initial release of the document.
Datasheet Volume 2 of 2
Introduction 1 Introduction The Intel® Xeon® Processor E7-8800/4800/2800 Product Families is the secondgeneration chip multiprocessor (CMP) offering Intel® QuickPath Interconnect (Intel® QPI) technology in the Intel® Xeon® MP processor family of processors. The Intel Xeon Processor E7-8800/4800/2800 Product Families implement up to ten multi-threaded (two thread) cores based upon the Intel Xeon Processor E7-8800/4800/2800 Product Families core design.
Introduction — Support for up to 16 DDR3 DIMMs per socket. Four DIMMs per Intel 7500 Scalable Memory Buffer — Support for DDR III 800, 978, 1067 MHz memory speeds — Support for 1, 2 and 4 Gigabit DRAM technology — Support for up to 32 GB Quad Rank DIMM — Support low voltage LV-RDIMMs (also called DDR3L) — Support for 1.5 V/1.
Introduction • Execute Disable Bit capability • Direct-attach firmware to processor socket via serial flash interface — Supports commodity 1-, 4-, 8-MB SPI Flash ROM devices 1.2 Terminology and Conventions This section defines the abbreviations, terminology, and other conventions used throughout this document. 1.2.1 Abbreviations Table 1-1.
Introduction Table 1-1. Abbreviation Summary (Sheet 2 of 3) Term 10 Description Intel® QuickPath Interconnect agent that handles IO IOH Input/Output Hub. An requests for processors.
Introduction Table 1-1. Abbreviation Summary (Sheet 3 of 3) Term Description WFS Wait for Startup Inter-Processor Interrupt (SIPI) XTPR External Task Priority MBox Integrated Memory Controller 1.3 Notational Conventions 1.3.1 Hexadecimal and Binary Numbers Base 16 numbers are represented by a string of hexadecimal digits followed by the character H (for example, F82EH). A hexadecimal digit is a character from the following set: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F.
Introduction 12 Datasheet Volume 2 of 2
Intel Xeon Processor E7-8800/4800/2800 Product Families Architecture 2 Intel Xeon Processor E7-8800/ 4800/2800 Product Families Architecture 2.1 Introduction The Intel Xeon Processor E7-8800/4800/2800 Product Families support up to ten-cores with up to 30-MB shared last-level cache (LLC) and two on-chip memory controllers. It is designed primarily for glueless four- or eight-socket multiprocessor systems, and features four Intel QuickPath Interconnects and four Intel SMI channels.
Intel Xeon Processor E7-8800/4800/2800 Product Families Architecture Table 2-1. Intel® Xeon® Processor E7-8800/4800/2800 Product Families and Intel® Xeon® Processor 7500 Series Key Features (Sheet 2 of 2) Intel® Xeon® Processor 7500 Series Features 2.1.
Intel Xeon Processor E7-8800/4800/2800 Product Families Architecture 2.2 Intel Xeon Processor E7-8800/4800/2800 Product Families Components (Boxes) The Intel Xeon Processor E7-8800/4800/2800 Product Families consist of ten Intel Xeon Processor E7-8800/4800/2800 Product Families cores connected to a shared, 30-MB inclusive, 30-way set-associative Last-Level Cache (LLC) by a high-bandwidth interconnect.
Intel Xeon Processor E7-8800/4800/2800 Product Families Architecture Figure 2-2.
Address Map 3 Address Map 3.1 NodeID Generation Intel Xeon processor 7500 series system addresses are made up of a socket and a device within the socket. With a 5-bit NodeID in the Intel QuickPath Interconnect SMP profile, Intel Xeon processor 7500 series can support up to four sockets (chosen by NID[3:2] when NID[4] is zero). Within each socket are four devices (NID[1:0]): IOH (00), B0/S0 (01), Ubox (10), B1/S1 (11).
Address Map 3.1.2 I/O Decoder Map Table 3-3 shows the I/O decoder address map. Given for each region are the name, the pattern for address bits [31:14], the size in bytes, the memory attribute, the number of targets in the target list, the address bits used to index the target list (if any), which CSR is used to enable the entry, and the location (decoder and entry number). For all regions which specify an address pattern, address bits [43:32] must be zero to match, except those marked with "*".
Address Map Table 3-3. I/O Decoder Entries (Sheet 2 of 2) Name Addr[31:14] Size Attr Tgts Index Enable Entry IntLog N/A N/A N/A 1 N/A always IOS6 IntPhy N/A N/A N/A 1 N/A always IOS6 EOI N/A N/A N/A 1 N/A always IOS6 FERR N/A N/A N/A 1 N/A always IOS5 Notes: 1. Non-contiguous In the Addr field, letters have the following meaning: • "x...
Address Map — Attempts to insert non-trusted VMM (rootkit hypervisor) Reset attacks designed to compromise platform secrets in memory — BIOS and firmware update attacks § 20 Datasheet Volume 2 of 2
LLC Coherence Engine (Cbox) and Caching Agent (Sbox) 4 LLC Coherence Engine (Cbox) and Caching Agent (Sbox) The Intel Xeon Processor E7-8800/4800/2800 Product Families core to the last level cache (LLC) interface is managed by the LLC coherence engine (Cbox). All Intel Xeon Processor E7-8800/4800/2800 Product Families core to Intel QuickPath Interconnect messages are handled through the Cbox and system interface logic.
LLC Coherence Engine (Cbox) and Caching Agent (Sbox) 4.1.1 LLC Major Features • Cache size: — 30 MB for ten Intel Xeon Processor E7-8800/4800/2800 Product Families core topologies • Organization: — Associativity: 24 ways — Line Size: 64 Bytes • Protection: — DECTED ECC for the data array — SECDED ECC correct for the tag array — SECDED ECC protection for the core valid array — SECDED ECC protection for the LRU array 4.
LLC Coherence Engine (Cbox) and Caching Agent (Sbox) Table 4-3. RTID Generation 6 LLC (Last Level Cache) Slices Sbox RTIDs Note: Freelist Lengths Base RTIDs 16 6, 5, 5 0, 6, 11 24 8, 8, 8 0, 8, 16 32 11, 11, 10 0, 11, 22 48 12, 12, 12 0, 12, 24 64 12, 12, 12 0, 12, 24 During power-on before Non-coherent (NC) freelist is available and programmed, only one RTID is available and core is not expected to get into C6 unless freelist is programmed.
LLC Coherence Engine (Cbox) and Caching Agent (Sbox) 24 Datasheet Volume 2 of 2
Home Agent and Global Coherence Engine (Bbox) 5 Home Agent and Global Coherence Engine (Bbox) Each Intel Xeon Processor E7-8800/4800/2800 Product Families home agent integrates a global coherence engine that is the center of the coherency activity for the cache lines owned by that home agent. The Bboxes receive Home channel request and snoop responses from caching agents, and provides data response and completion to the system via Bbox to Router connection.
Home Agent and Global Coherence Engine (Bbox) 5.1.1 The Tracker Modes The tracker modes (0..3) are specifically chosen to support systems consisting of 4 gluelessly connected Intel Xeon Processor E7-8800/4800/2800 Product Families sockets and 2, 4 IOH/XNC nodes (IOH = I/O Hub, XNC = eXternal Node Controller). Minimal support is also provided for 8 IOH/XNCs. Mode 5 has been added to support glueless 8 Intel Xeon Processor E7-8800/4800/2800 Product Families sockets and 4 IOH nodes.
Home Agent and Global Coherence Engine (Bbox) Table 5-2. TID Assignment Restrictions (Sheet 2 of 2) Mode 5 Configuration 8S HemiSphere + 4 IOH/XNC Sboxes IOH/XNCs TID [0...31],TID[0..15] TID [0...15] 6 4S Hemisphere + 2 IOH/XNC TID [0…63], TID [0…31] TID [0...31] 7 4S Non-Hemisphere + 2 IOH/XNC TID [0…31], TID [0…15] TID [0...31] Note: TID Assignment Restrictions have been updated for modes 4, 5, 6, and 7. 5.2 Directory Assisted Snoopy (DAS) DAS stands for Directory Assisted Snoopy.
Home Agent and Global Coherence Engine (Bbox) IODC is supported only in 4 socket configurations with 2 IOH or less. It is also not supported with 8 socket tracker modes 4 and 5. IODC and DAS features are orthogonal. They can not be supported together. DAS targets memory latency improvements for snoop bound topologies and IODC targeting streams memory bandwidth improvement. 4/2 sockets topologies are not expected to benefit from DAS since the latency is memory bound.
System Configuration Controller (Ubox) 6 System Configuration Controller (Ubox) The Intel Xeon Processor E7-8800/4800/2800 Product Families contain a system configuration controller (Ubox). Figure 6-1 provides a Intel Xeon Processor E7-8800/4800/2800 Product Families block diagram including the Ubox. Figure 6-1.
System Configuration Controller (Ubox) 30 Datasheet Volume 2 of 2
Memory Controller (Mbox) 7 Memory Controller (Mbox) The Intel Xeon Processor E7-8800/4800/2800 Product Families consists of two integrated memory controllers. The memory controller (Mbox) contains the interface logic to Intel 7500 Scalable Memory Buffer via Intel SMI interface (formerly “Fully Buffered DIMM 2 interface”). Mbox issues memory read and write commands per Intel SMI protocol and schedules them with respect to DDR III timing.
Memory Controller (Mbox) Note: Memory Mirroring with tracker mode 6 is not supported. Note: RFR_FSM errors may be logged in the Mbox while in 2x refresh. 7.3 Double Device Data Correction (DDDC) DDDC (Double Device Data Correction) is a feature which assures data availability after hard failure of 2 x4 DRAM’s. Supports x4 DDDC plus an additional single bit error correction. 7.3.1 DDDC flow overview • One x4 DRAM device in each rank is reserved as spare device.
Memory Controller (Mbox) 7.5 Partial Memory Mirroring Partial memory mirroring is a mirror mode of operation with parts of system memory operating with redundant memory at slave, leaving the rest of the system memory in non-mirror mode. The granularity of the partial memory that gets to operate in mirror mode is implementation dependent.
Memory Controller (Mbox) Figure 7-1.
Memory Controller (Mbox) 7.5.2.1 Pre-conditions • Assumes Master/Slave with identical memory types, and density. • Assumes existing configuration limitations applicable to full mirror mode of operation. 7.5.2.2 Sample Scenario • System manufacturer with tight control on OS kernel will configure the server platform in partial memory mirroring mode. • System configuration (SAD entries) is such that the software kernels will be loaded in physical Home Agent/memory.
Memory Controller (Mbox) characteristics like power consumed/saved, entry latency, exit latency. Memory contents may be retained or lost by a power state. A power state may be controlled by hardware or software or hybrid. A memory power node can be placed at various power states depending on the power savings versus latency trade off. the above power state characteristics are vital for OS/VMM to make this trade off. These numbers are also provided to OS by BIOS via the ACPI MPST Table. 7.7.
Physical Layer (Pbox) 8 Physical Layer (Pbox) The Intel Xeon Processor E7-8800/4800/2800 Product Families have two fully buffered DIMM (FBD) ports; each port is of two FBD channels connected to the Buffer-On-Board (BoB), the Intel 7500 Scalable Memory Buffer. There are four full-width Intel QPI ports for inter-processor communications. The Physical layer for these internal (FBD) and external (Intel QPI) channels/links is implemented in Pbox.
Physical Layer (Pbox) Mbox M0 Bbox B0 Sbox S0 Sbox S1 Port-2 Port-6 Port-3 Pbox PZ0 Intel Xeon Processor E7-8800/4800/2800 Product Families System Interface Port-1 Pbox PR0 Port-7 Figure 8-1. RBox Port-0 Port-4 Pbox PR1 Pbox PR2 Port-5 Bbox B1 Mbox M1 Pbox PZ1 Ubox UU Pbox PR3 Figure 8-1 shows the interface of each of Pbox instances with other uncore boxes. PZ0 and PZ1 are the Pbox-FBD port instances and PR0 to PR3 is the Pbox-Intel QPI port instances.
Power Management Architecture (Wbox) 9 Power Management Architecture (Wbox) The power management control unit, (Wbox), controls power management functionality based on the current behavior and desired operating point for each of the cores. Each core provides information on the desired power state of that core, core temperature information, voltage seen by the core and desired operating frequency to the Wbox. The Wbox is responsible for power management functions including: 1.
Power Management Architecture (Wbox) 9.1.3 THERMTRIP# All thermal sensors, including the uncore thermal sensor, have a catastrophic trip output which is asserted when sensor temperature exceeds its thermtrip threshold temperature. These signals are all asynchronously or'd together onto the THERMTRIP# pin. Assertion of any of the catastrophic trip signals causes disabling of all the PLLs through internal powergood de-assertion for quick response.
Power Management Architecture (Wbox) referred to as ''C'' states. C0 refers to the processor active state, and all other C-states are idle states. Higher numbered C-states are lower power, but longer latency. C3 is lower power than C1, and so on. States C0, C1 require that processor caches maintain coherence – in other words, they must ensure that any memory requests from other system agents receive the latest copy of the data if it is stored in the processors cache.
Power Management Architecture (Wbox) Figure 9-1. Valid Thread/Core Architectural C-State Transitions C0 MWAIT C1, HLT MWAIT ,C6 I/O C6 2 2 2 HALT/ MWAIT C1E 1 MWAIT C3, I/O C3 C1E C1 2 C3 C6 1. No transition to C 0 is needed to service a snoop when in C 1 or C1E. . 2. Transitions back to C 0 occur on an interrupt or on access to monitored address 9.2.2.1.1 (if state was entered via MWAIT ). .
Power Management Architecture (Wbox) 9.2.2.2 Package C-State Resolution The package must resolve the C-state requests of each core in order to determine the proper package C-state. The C-state request is resolved by the Wbox to the lowest numbered C-state requested by any of the enabled cores. If any core is in C0, the resolved package C-state is C0. If no cores are in C0, and at least one core is in C1, then C1 is the resolved package C-state, and so on. See the following table. Table 9-2.
Power Management Architecture (Wbox) If this is the last enabled thread to enter the C3 or lower state, the Intel Xeon Processor E7-8800/4800/2800 Product Families flushes the I cache, D cache, and MLC before putting the thread to sleep. Note that these flush operations are atomic – if a break event to either thread occurs after the flush of a cache is begun, the flush will complete. 9.2.2.4.2 Thread/Core C3 Exit When the core wakes up, the core active bit in the Wbox is set by core hardware.
Power Management Architecture (Wbox) defaults to having a sub-state of zero. However, I/O redirected MWAITs always assumes the 'break on IF=0' control that can be selected using ECX=1 with an MWAIT instruction. 9.2.2.7 Core C3 Auto Demote The operating system requests entry to a specific C-state by supplying the appropriate hint with the MWAIT instruction.
Power Management Architecture (Wbox) 9.4.1 Introduction The PMReq negotiation is done to enter package C6 state where uncore power optimization actions are taken.The package will attempt to enter the C6 state when all cores have transitioned to the C6. Once the package has entered the C6 state, it will only be woken when it receives a break event, memory transaction or a snoop. It also wake up from macro-clock gating for some of the PECI transactions which requires uncore clocks.
Power Management Architecture (Wbox) 9.5.2 PMReq Retry/CmpD Response Behavior 9.5.2.1 PMReq Retry Determination Retries are the re-querying to other sockets to see if it this node can make transition to its desired package C-state. Retries are done when it is found that system state is changed which may cause previous responses from other sockets stale.
Power Management Architecture (Wbox) a. Messages initiated in response to a PMReq message from another node that indicates the deepest state that this node will permit the originating node to enter. 3. Inbound PMReq messages: a. Requests from other node(s) for permission to transition to a deeper state, or b. Announcements from other nodes that they have transitioned to a shallower state. 4. Inbound CmpD messages: a. Responses from other node(s) to a PMReq message from this node. 9.
Power Management Architecture (Wbox) Intel Xeon Processor E7-8800/4800/2800 Product Families expects mailbox sideband limit request as core clock multiplier ratio corresponding to a valid P-state defined in ACPI table (ACPI table is visible to PECI Host Controller). The pcode support for mailbox interface will be designed to accept all possible P-states (clock ratios) limit requests, it will allow design flexibility to add any new p-state between P1 and Pn.
Power Management Architecture (Wbox) 50 Datasheet Volume 2 of 2
