«Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture Oracle White Paper—Oracle's Sun SPARC ...»
Floating point/graphics unit. A floating point/graphics unit (FGU) is provided within each core and it is shared by all eight threads assigned to the core. Thirty-two floating-point register file entries are provided per thread.
Stream processing unit. Each core contains a stream processing unit (SPU) that provides cryptographic coprocessing.
Memory management unit. The memory management unit (MMU) provides a hardware table walk (HWTW) and supports 8 KB, 64 KB, 4 MB, and 256 MB pages.
An eight-stage integer pipeline and a 12-stage floating-point pipeline are provided by each UltraSPARC T2 and UltraSPARC T2 Plus processor core (Figure 10). A new “pick” pipeline stage has been added to choose two threads (out of the eight possible per core) to execute each cycle.
Figure 10. An 8-stage integer pipeline and a 12-stage ﬂoating-point pipeline are provided by each UltraSPARC T2 and UltraSPARC T2 Plus processor core.
To illustrate how the dual pipelines function, Figure 11 depicts the integer pipeline with the load store unit (LSU). The instruction cache is shared by all eight threads within the core. A leastrecently-fetched algorithm is used to select the next thread to fetch. Each thread is written into a thread-specific instruction buffer (IB) and each of the eight threads is statically assigned to one of two thread groups within the core.
Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture Figure 11. Threads are interleaved between pipeline stages with very few restrictions (integer pipeline shown, letters depict pipeline stages, numbers depict different scheduled threads).
The pick stage chooses one thread each cycle within each thread group. Picking within each thread group is independent of the other, and a least-recently-picked algorithm is used to select the next thread to execute. The decode state resolves resource conflicts that are not handled during the pick stage. As shown in the illustration, threads are interleaved between pipeline stages with very few restrictions. Any thread can be at the fetch or cache stage, before being split into either of the two thread groups. Load/store and floating-point units are shared between all eight threads. Only one thread from either thread group can be scheduled on such a shared unit.
By providing integrated on-chip networking, the UltraSPARC T2 processor is able to provide better networking performance. All network data is supplied directly from and to main memory.
Placing networking so close to memory reduces latency, provides higher memory bandwidth, and eliminates inherent inefficiencies of I/O protocol translation.
The UltraSPARC T2 processor provides two 10 GbE ports with integrated serializer/deserializer (SerDes), offering line-rate packet classification at up to 30 million packets/second (based on layers 14 of the protocol stack). Multiple DMA engines (16 transmit and 16 receive DMA channels) match DMAs to individual threads, providing binding flexibility between ports and threads. Virtualization support includes provisions for eight partitions, and interrupts may be bound to different hardware threads.
Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture Stream Processing Unit The SPU on each UltraSPARC T2 core runs in parallel with the core at the same frequency. The cipher/hash unit provides support for popular RC4, DES/3DES, AES-128/192/256, MD5, SHA-1, and SHA-256 ciphers. The SPU is designed to achieve wire-speed encryption and decryption on both of the processor’s 10 GbE ports.
Integral PCI Express Support
The UltraSPARC T2 and UltraSPARC T2 Plus processors provide an on-chip PCIe interface that operates at 4 GB/sec bidirectionally through a point-to-point dual-simplex chip interconnect. An integral IOMMU supports I/O virtualization and process device isolation by using the PCIe BUS/Device/Function (BDF) number. The total I/O bandwidth is 3 GB/Sec to 4 GB/sec, with maximum payload sizes of 128 to 512 bytes. An x8 SerDes interface is provided for integration with off-chip PCIe switches.
Beyond the inherent efficiencies of CMT design, the UltraSPARC T2 and UltraSPARC T2 Plus processors are the first to incorporate unique power management features at both the core and memory levels of the processor. These features include reduced instruction rates, parking of idle threads and cores, and ability to turn off clocks in both cores and memory to reduce power consumption. Substantial innovation is present in the areas of Limiting speculation such as conditional branches not taken Extensive clock gating in the data path, control blocks, and arrays Power throttling that allows extra stall cycles to be injected into the decode stage Server Architecture Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers have been designed to provide breakthrough performance while maximizing reliability and minimizing power consumption and complexity. This section details the physical and architectural aspects of these systems.
System-Level Architecture The system-on-a-chip (SoC) design of the UltraSPARC T2 and T2 Plus processors mean that sophisticated system-level functionality can be accomplished with a minimum of high-quality components. The sections that follow describe the architecture of the various systems.
Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture Oracle's Sun SPARC Enterprise T5120/T5220 Servers A unified motherboard design is common to both the Oracle Sun SPARC Enterprise T5120/T5220 servers (Figure 12).
Figure 12. A uniﬁed motherboard design is common to both Oracle's Sun SPARC Enterprise T5120/T5220 servers.
The motherboard is a 20-layer printed circuit board (PCB) containing the UltraSPARC T2 processor, FB-DIMM sockets for main memory, Integrated Lights Out Manager service processor, disk controller, and I/O subsystems. I/O options include USB, DVD control, quad Gigabit Ethernet, and two levels of PLX PCIe branching out into sockets for a wide variety of third-party PCIe expansion options. Shaded regions indicate features that are only available on Oracle's Sun SPARC Enterprise T5220 server.
Oracle's Sun SPARC Enterprise T5140/T5240 Servers Oracle's Sun SPARC Enterprise T5140/T5240 servers share a common motherboard design (Figure 13).
Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture Figure 13. Oracle's Sun SPARC Enterprise T5140/T5240 servers share a common motherboard design.
Key features of the Oracle Sun SPARC Enterprise T5140/T5240 motherboards include Dual sockets for UltraSPARC T2 Plus processors, connected by four coherence links A memory mezzanine tray to supply additional memory to Oracle's Sun SPARC Enterprise T5240 server (up to 256 GB system maximum with 8 GB low-voltage FB-DIMMs) Integration of Oracle’s Neptune chip to provide 10 Gigabit Ethernet functionality as well as standard quad Gigabit Ethernet ports (10/100/1000-BaseT) The motherboard interconnects for all of these systems have been greatly simplified over previous-generation systems. Twelve-volt power is distributed to the motherboard through a pair of metal bus bars, connected to a power distribution board (PDB). A single flex circuit connector routes all critical power control and DVD drive signaling over to the PDB. One or two mini-SAS cables connect the motherboard to the disk drive backplane, providing data access to the system hard drives.
Memory Subsystem Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture In Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers, the UltraSPARC T2 or UltraSPARC T2 Plus processor provides on-chip memory controllers that communicate directly to FB-DIMM memory through high-speed serial links. Four dual-channel FBDIMM memory controller units (MCUs) are provided on the UltraSPARC T2 processor while the UltraSPARC T2 Plus processor provides two MCUs. Each MCU can transfer data at an aggregate rate of 4.0 Gb/sec (UltraSPARC T2) or 4.8 Gb/sec (UltraSPARC T2 Plus). Sixteen motherboard memory socket locations on Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers provide sufficient board space for two rows of 667 MHz FB-DIMMs per channel. Oracle's Sun SPARC Enterprise T5240 servers support an optional memory mezzanine tray that can be added to support an additional 16 FB-DIMM slots. The memory mezzanine tray (Figure 14) allows Oracle's Sun SPARC Enterprise T5240 server to support up to 256 GB of RAM using 8 GB FBDIMMs.
Figure 14. An optional memory mezzanine tray doubles the memory capacity of Oracle's Sun SPARC Enterprise T5240 servers to up to 256 GB.
I/O Subsystem Each UltraSPARC T2 and UltraSPARC T2 Plus processor incorporates a single, eight-lane (x8) PCIe port capable of operating at 4 GB/sec bidirectionally. In each server, this port natively interfaces to the I/O devices through a series of PLX technology PCIe expander chips, connecting either to PCIe card slots, or to bridge devices that interface with PCIe, such as those listed below.
Disk controller. Disk control is managed by an LSI Logic SAS1068E SAS controller chip that interfaces to a four-lane (x4) PCIe port. RAID levels 0 and 1 are provided as standard.
Modular disk backplanes. Depending on the system, a 4-, 8-, or 16-disk backplane is attached to the LSI disk controller by one or more x4 SAS links. The 16-disk backplane provides a 28-port LSI Logic SAS Expander to support the additional disk drives.
Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture Solid state drives. Solid state drives (SSDs) can be substituted for disk drives in all Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers. Servers in a 1RU chassis such as Oracle's Sun SPARC Enterprise T5120/T5140 servers can support up to four SSDs. Servers in a 2RU chassis such as Oracle's Sun SPARC Enterprise T5220/T5240 servers can support up to eight SSDs. Remaining slots can be occupied with conventional SAS hard disk drives.
Gigabit Ethernet. On Oracle's Sun SPARC Enterprise T5120/T5220 servers, two x4 PCIe ports connect to two Intel Ophir dual Gigabit Ethernet chips, providing four 10/100/1000 Mb/sec Ethernet interfaces on the rear of each chassis. On Oracle's Sun SPARC Enterprise T5140/T5240 servers, Oracle’s Neptune Ethernet chip provides two 10/100/1000-BaseT ports and two 10/100/1000/10000-BaseT interfaces, exposed as four RJ-45 connectors on the rear panel.
Dual 10 Gigabit Ethernet. Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers all provide dual 10 Gb X (ten) Attachment Unit Interface (XAUI) connections, expressed through shared XAUI/PCIe slots. On Oracle's Sun SPARC Enterprise T5120/T5220 servers, these ports are provided by the dual 10 GbE ports integrated into the UltraSPARC T2 processor. On Oracle's Sun SPARC Enterprise T5140/T5240 servers, the 10 GbE interfaces are provided by Oracle’s Neptune Ethernet chip.
When the 10 GbE ports are connected, two of the Gigabit Ethernet ports become unavailable for use.
USB and DVD. On all servers, a single-lane PCIe port connects to a PCI bridge device. A second bridge chip converts the 32-bit 33 MHz PCI bus into multiple USB 2.0 ports. The system’s USB interconnect is driven from those ports. On Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers, the DVD is driven from a further bridge chip that interfaces one of the USB ports to IDE format.
Chassis Design Innovations Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers share basic chassis design elements. This approach not only provides a consistent look and feel across the product line, but it simplifies administration through consistent component placement and shared components. Beyond mere consistency, this approach reflects a data center design focus that places key technology where it can make a difference for the data center.
Enhanced system and component serviceability. Finding and identifying servers and components in a modern data center can be challenging. Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 servers are optimized for lights-out data center configurations with easy-to-identify servers and modules. Color-coded operator panels provide easy-to-understand diagnostics and systems are designed for Oracle White Paper—Oracle's Sun SPARC Enterprise T5120/T5220 and Oracle's Sun SPARC Enterprise T5140/T5240 Server Architecture deployment in hot-isle/cold-isle multiracked deployments with both front and rear diagnostic LEDs to pinpoint faulty components. Fault Remind features identify failed components.
Consistent connector layouts for power, networking, and management make moving between Oracle’s systems straightforward. All hot-plug components are tool-less and easily available for serviceability. For instance, an integral hinged lid provides access to dual fan modules so that fans can be serviced without exposing sensitive components or causing unnecessary downtime.
Robust chassis, component, and subassembly design. Oracle’s volume servers share chassis that are carefully designed to provide reliability and cool operation. Even features such as the hexagonal chassis ventilation holes are designed to provide the best compromise for high strength, maximum airflow, and maximum electronic attenuation. Next-generation hard disk drive carriers leverage the hexagonal ventilation of the chassis and provide a 7 percent smaller front plate for greater storage density while increasing airflow to the system.