For the past five to seven years the most common speed of connection between a Top of Rack (ToR) leaf switches to subtending computer and storage servers has been 10Gbps. Many hyperscale data centers and even larger enterprise data centers are migrating these access links to 25Gbps. These connections may be constructed using 25Gbps Direct Attached Copper cables (DACs), Active Optical Cables (AOCs), or with a pair of SFP28 25Gbps optical transceivers and an appropriate duplex fiber optic jumper cable.
To depict this application with real-world products, a ToR switch from Cisco’s Nexus 3000 series and a rack-mounted server from Supermicro have been selected. The only other pieces required are FluxLight SFP-25G-SR-s multimode transceivers and OM4 multimode patch cables.
TOR LEAF SWITCH: Cisco Nexus 34180YC
The Nexus 3400 platform is part of the latest generation of the fixed Nexus®3000 Series Switches. The 3000 series is squarely targeted for ToR applications. All members are compact (1RU) fixed configuration products. Across this family of products essentially all available rates of optical ethernet are offer, from 1G to 400G.
The Nexus 34180YC is an ideal switch to demonstrate the used of the FluxLight brand, Cisco compatible SFP-25G-SR-S transceiver. This switch offers great flexibility in ports speeds, covering 1G, 10G, 25G, 40G and 100G rates. The 34180YC is programmable allowing the user to customers to tailor packet forwarding behavior to the needs of their applications. For example, high speed financial trading applications can be optimized for the lowest possible latency. This switch is equipped with 48 SFP+/SFP28 ports (1G/10G/25G) and 6 QSFP+/QSFP28 (40G/100G) ports. The switch supports full line rate Layer 2/3 switching on all these ports, totaling 3.6 Terabits/sec and 1.4 Gigapackets/sec.
The 34180YC may be equipped with a wide range of optical transceiver interface types across the many rates mentioned above. The tables below include the compatible transceiver types for the two categories of ports in the switch.
Table 3 SFP+/SFP28 Optical Transceivers
| SFP-10G-SR | SFP+ SR, 300m/OM3, 400m/OM4, duplex LC |
| SFP-10G-LR | SFP+ LR, 10km, SMF, duplex LC |
| SFP-10G-LRM | SFP+ LRM, 220 OM1 MMF, duplex LC |
| SFP-10G-ER | SFP+ ER, 40km, SMF, duplex LC |
| SFP-10G-ZR | SFP+ ZR, 80km, SMF, duplex LC |
| SFP-10G-T | SFP+ T, 30m, CAT7, RJ45 |
| SFP-10GBX-UD-PAIR | SFP+ BX, 10km, SMF, simplex LC |
| SFP-10GBX-U-40, | SFP+ BX, 40km, SMF, simplex LC |
| SFP-25G-SR-S | SFP28 SR, 70m/OM3, 100m/OM4, duplex LC |
| SFP-25G-LR -S | SFP28 LR, 10km, SMF, duplex LC |
Table 4 QSFP+/QSFP28 Optical Transceivers
| QSFP-40G-SR4 | QSFP+ 40G SR4, 100m, OM3 MMF, MPO connector |
| QSFP-40G-CSR4 | QSFP+ 40G CSR4, 300m OM3 MMF, MPO connector |
| WSP-Q40GLR4L | QSFP+ 40G LR4L, 2Km SMF, duplex LC |
| QSFP-40G-ER4 | QSFP+ 40G ER4, 40Km SMF, duplex LC |
| QSFP-40G-SR-BD | QSFP+ 40G Bidirectional MMF, 100m OM3, duplex LC |
| QSFP-4X10G-LR-S | QSFP+ 40G, 10km, SMF, MPO connector |
| QSFP-40G-LR4 | QSFP+ 40G LR4, 10km SMF, duplex LC |
| QSFP-100G-SR4-S | QSFP28 100G SR4, 100m, OM3 MMF, MPO connector |
| QSFP-100G-CWDM4-S | QSFP28 100G PSM4, 500m SMF, MPO connector |
| QSFP-100G-PSM4-S | QSFP28 100G PSM4, 2Km SMF, duplex LC |
| QSFP-100G-ER4L-S | QSFP28 100G ER4, 40Km SMF, duplex LC |
| QSFP-100G-LR4-S | QSFP28 100G LR4, 10km SMF, duplex LC |
Rack-mounted Server: SuperMicro SYS-1029U-TR25M
Supermicro’s Ultra line of servers are aimed at virtualization, cloud computing and high-end enterprise server applications. Key features of the 1029U-TR25M are:
- Dual socket P supporting Intel XEON scalable processors
- 2 PCI-E 3.0 x16 slots; 1 PCI-E 3.0 x8 slot; 1 PCI-E x16 slot (internal LP)
- Up to 3TB of ECC 3DS LRDIMM or RDIMM, up to DDR4-2666MHz; 24 DIMM slots
- 10 Hot-swap 2.5” drive bays, 10 SATA3; optional M.2 NVMe and SATA3 ports
- 8 heavy-duty fans
- 750W Redundant Power Supplies
- 2x 25GbE SFP28 slots
25GBASE-SR OPTICAL TRANSCEIVERS: FluxLight SFP-25G-SR-S
FluxLight’s SFP-25G-SR (click link for datasheet) is a Cisco® compatible 25GBASE-SR SFP+ Optical Transceiver and is factory pre-programmed with all the necessary configuration data for fully compatible operation with all Cisco equipment offering SFP+ slots. The FluxLight SFP-25G-SR includes an dual-LC connector and supports links of up to 100 meters on OM4 multimode fiber.
This device performs identically to Cisco® original transceivers, are 100% compatible and support full hot swappable operation. The SFP-10G-T is 100% MSA (Multi-Source Agreement) compliant, see: SFP+ MSA,
OM4 MULTIMODE JUMPER CABLE: FluxLight PN: FL-LCLC-OM4-2M
To connect the SFP-25G-SR-S modules in the Cisco Nexus 34180YC and the Supermicro server SFP28 ports a duplex LC-LC multimode OM4 fiber jumper is used. Typically a short jumper is connected, one end to the SFP28 module and the other to a nearby patch panel.
Putting it all together
The Cisco Nexus 34180YC switch occupies 1RU and is typically placed at the top of the rack. Each Supermicro server is also 1RU high. In this example two 25Gbps links are used per server, therefore, two FluxLight SFP-25G-SR-S modules must be installed in the 34180YC for each server. So, in the depiction below, a total of 8 FluxLight SFP-25G-SR-S modules are used.
Each Nexus 34180, with its 48 SFP+/SFP28 ports, supports up to 48 servers. Since the servers are dual-homed, in this example at 25Gbps, the total uplink capacity per server is 50Gbps. If any single link or leaf switch fails, the uplink capacity still continues at 25Gbps.
Cisco switches were used to demonstrate the application. FluxLight carries 25GBASE-SR modules compatible with a wide range of OEM switches. Some popular models are listed below.
Table 5 SFP Optical Transceivers for Popular Switch OEMs
| SFP-25G-SR | Arista |
| 25G-SFP28-SR | Brocade-Foundry |
| 10501 | Extreme |
| SFP28-25G-SR | Dell |