Example 2: Connecting Cisco Nexus 9336C-FX2 40G QSFP+ ports to HPE ProLiant DL360 Gen10 Rack Servers Equipped with HPE 530SFP+ 10GbE 2-port Network Adapters.
Continuing in our series on options for the connection of a leaf switch to rack-mounted servers, we are going to step up the complexity to achieve higher connectivity density. In our previous post (see: https://blog.fluxlight.com/2018/10/09/cisco-nexus-93180yc-fx-to-dell-poweredge-r640/) connecting the leaf switch to the servers were simple 10Gbps SFP+-to-SFP+ (point to point) links using OM3 multimode fiber. In today’s post we will use 40Gbps QSFP+ ports on the leaf switch end of the connection. A point-to-multipoint connection will be used to allow each QSFP+ port to connect to four separate SFP+ ports on the servers.
In the previous post, four SFP+ ports in the Nexus 93180YC-FX switch supported connections to two servers, each connected using two 10Gbps links. By using the 40Gbps ports on the Nexus 9336C, a single QSFP+ slot supports two servers…a 4:1 reduction in the number of switch ports.
TOR Leaf Switch: Cisco Nexus 9336C-FX2
The Nexus® 9300-FX platform is part of the latest generation of the fixed Nexus® 9000 Series Switches. Cisco has coined the phrase “Cisco Cloud Scale (CCS)” to describe the technology on which this series is based. The primary elements of CCS are,
- Greater performance scaling using multi-rate ports (1/10/25/50/100G)
- Lower Total Cost of Ownership
- Real-time telemetry and analytics with greater detail
- Improved security through line-rate encryption
- Faster application completion time
The Cisco Nexus 9336C-FX2 Switch is a 1RU switch that supports 7.2 Tbps of bandwidth and over 2.8 bpps (billion packets per second). This particular model is easily the most flexible of the Nexus 9300 line. Each of the switch’s 36 ports can be configured to work as 1/10/25/40/100-Gbps. The ports natively support QSFP+ (40Gbps) and QSFP28 (100Gbps) modules. 1G (SFP), 10G (SFP+) and 25G (SFP28) are supported using adaptors called QSAs. All ports support full line rate encryption (MACsec). Breakout, 40G to 4x10G and 100G to 4x25G, is supported on all ports.
Rack-mounted Server: HPE ProLiant DL360 Gen10 Server
HPE claims, “The powerful 2P HPE ProLiant DL360 Gen10 is redefining dense compute by delivering security, agility and unmatched expandability businesses want all packed in a dense 1U rack design.” The ProLiant DL360 Gen10 Server is based Intel® Xeon® processor(s) and can be ordered with up to 28 cores, plus HPE DDR4 SmartMemory operating at 2.666 Billion Transfers/Second (GT/s) supporting up to 3.0 TB maximum memory space. To further boost performance, the Non-Volatile Memory of the HPE ProLiant DL360 Gen10 includes 12 NVDIMMs and 10 NVMe (SSD). This platform “supports diverse workloads in space constrained environments and maintains it with ease by automating the most essential server lifecycle management tasks with HPE OneView and HPE iLO 5.”
Dual-SFP+ Port 10G Adaptor: HPE 530SFP+ Network Adaptor
The HPE 530SFP+ is an Ethernet adapter with 2 SFP+ 10Gbps ports in a PCIe 2.0 compliant form factor. It is compatible with select HPE ProLiant Gen10, Gen9, Gen8 and Gen7 ML/DL/SL servers. The HPE 530SFP+ combines full line-rate performance across both ports with low power consumption (less than 8 watts max). Ethernet transfer rate of 20 Gbps full duplex per port (40 Gbps aggregate) improves response times and alleviates bottlenecks in next generation data centers. The adaptor supports enterprise class features such as VLAN tagging, adaptive interrupt coalescing, MSI-X, NIC teaming (bonding), Receive Side Scaling (RSS), jumbo frames, PXE boot and virtualization features such as VMware NetQueue and Microsoft VMQ.
10GBASE-SR Optical Transceivers: FluxLight QSFP-40G-SR4 and FluxLight J9150A
FluxLight’s QSFP-40G-SR4 is a Cisco® compatible 40GBase-SR4 SFP+ Optical Transceiver and is factory pre-programmed with all the necessary configuration data for seamless network integration. It supports 4 lanes of 10Gbps Ethernet transmission over up to 150 meters of OM3 multimode fiber and transmits at 850nm. The four 10Gbps lanes each travel over 2 fibers in an MPO-12 cable (4 unused fibers). These transceivers perform identically to Cisco® original transceivers and are 100% compatible and support full hot swappable operation. The QSFP-40G-SR4 is 100% MSA (Multi-Source Agreement) compliant, see: QSFP+ MSA, SFF-8436.
FluxLight’s J9150A is fully compliant with the SFP+ MSA (SFF-8431) and backed by Fluxlight’s limited lifetime warranty. The primary difference between the SFP-10G-SR (highlighted in our previous post) and the J9150A is compatibility. The J9150 is an HPE® compatible 10GBase-SR SFP+ optical transceiver. It is pre-programmed by Fluxlight with all the necessary configuration data for seamless integration into all supporting HPE equipment. These transceivers perform identically to HPE® original transceivers and support hot swappable operation.
Jumper Cables: MPO to 4 x Dual LC OM3 Multimode Breakout Cable
To connect the QSFP+ ports in the Cisco Nexus 9336C-FX2 to the 530SFP+ network adaptor modules in in the ProLiant DL360, an MPO to 4x Dual-LC breakout cable is used. The MPO end of the cable is inserted into the Fluxlight QSFP-40G-SR4. Each of the Dual-LC connectors are inserted into one the Fluxlight J9150A SFP+ modules which are installed in the HPE 530SFP+ network adaptors installed in the ProLiant DL360 servers.
Putting it all together
We’ve listed all the components so now we need to connect the bits together. The Cisco Nexus 9336C-FX2 switch only uses 1RU and is customarily placed at the top of the rack. Each HPE ProLiant DL360 server also uses only 1RU. Since we are running two 10Gbps links per server, two Fluxlight 10GBASE-SR modules must be installed in each of the network adaptor cards (530SFP+). In this case the optical transceivers must HPE compatible, so the correct Fluxlight part number is J9150A.
The Nexus 9336C-FX2 switch contains 36 multi-rate slots. Assuming not all of the traffic is locally connecting between the local ProLiant servers, we will set aside enough ports that, when equipped with 100G QSFP28 modules (for example, FluxLight QSFP-100G-SR4) will support approximately all the traffic to/from the subtending servers (downlinks). If we allocate 26 ports to 40G QSFP+ connections to servers, that is a total of 1.04 Terabits/sec (Tbps) downlink capacity. Equipping the remaining 10 ports with 100G QSFP28 optical transceivers, brings the total uplink capacity to 10 x 100 or 1.0 Tbps, so just about a perfect split. Of course this is a bit simplistic. The actual allocation may involve many other factors including things like how many spine switches are used in the center design, predictions of intra-rack traffic, etc.
Finally, the MPO to 4x Dual-LC OM3 duplex multimode fiber jumper mentioned above, is used to connect to each QSFP+ module in the Nexus leaf switch to four SFP+ modules in the server NICs. Each QSFP+ module supports two servers with a total of 20Gbps (full-duplex) each. Since we have allocated 26 Nexus 9336C ports to 40G QSFP+ modules, that brings the total number of servers per 9336C switch to 52. In our previous post, using the 48 SFP+ 10Gbps ports on the Cisco Nexus 93180YC-FX, allowed only 24 subtended servers, so we have more than doubled the leaf-server density by using the 9336C-FX2 QSFP+ ports.
Available at FluxLight
Buy Now: FluxLight QSFP-40G-SR4
Buy Now: FluxLight J9150A