100 Gigabit Ethernet

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

40 Gigabit Ethernet (40GbE) and 100 Gigabit Ethernet (100GbE) are groups of computer networking technologies for transmitting Ethernet frames at rates of 40 and 100 gigabits per second (Gbit/s), respectively. The technology was first defined by the IEEE 802.3ba-2010 standard[1] and later by the 802.3bg-2011, 802.3bj-2014,[2] and 802.3bm-2015 standards.[3]

The standards define numerous port types with different optical and electrical interfaces and different numbers of optical fiber strands per port. Short distances (e.g. 7 m) over twinaxial cable are supported while standards for fiber reach up to 40 km.

History[edit]

Standards Development[edit]

On July 18, 2006, a call for interest for a High Speed Study Group (HSSG) to investigate new standards for high speed Ethernet was held at the IEEE 802.3 plenary meeting in San Diego.[4]

The first 802.3 HSSG study group meeting was held in September 2006.[5] In June 2007, a trade group called "Road to 100G" was formed after the NXTcomm trade show in Chicago.[6]

On December 5, 2007, the Project Authorization Request (PAR) for the P802.3ba 40 Gbit/s and 100 Gbit/s Ethernet Task Force was approved with the following project scope:[7]

The purpose of this project is to extend the 802.3 protocol to operating speeds of 40 Gbit/s and 100 Gbit/s in order to provide a significant increase in bandwidth while maintaining maximum compatibility with the installed base of 802.3 interfaces, previous investment in research and development, and principles of network operation and management. The project is to provide for the interconnection of equipment satisfying the distance requirements of the intended applications.

The 802.3ba task force met for the first time in January 2008.[8] This standard was approved at the June 2010 IEEE Standards Board meeting under the name IEEE Std 802.3ba-2010.[9]

The first 40 Gbit/s Ethernet Single-mode Fibre PMD study group meeting was held in January 2010 and on March 25, 2010 the P802.3bg Single-mode Fibre PMD Task Force was approved for the 40 Gbit/s serial SMF PMD.

The scope of this project is to add a single-mode fiber Physical Medium Dependent (PMD) option for serial 40 Gbit/s operation by specifying additions to, and appropriate modifications of, IEEE Std 802.3-2008 as amended by the IEEE P802.3ba project (and any other approved amendment or corrigendum).

On June 17, 2010, the IEEE 802.3ba standard was approved [1][10] In March 2011 the IEEE 802.3bg standard was approved.[11] On September 10, 2011, the P802.3bj 100 Gbit/s Backplane and Copper Cable task force was approved.[2]

The scope of this project is to specify additions to and appropriate modifications of IEEE Std 802.3 to add 100 Gbit/s 4-lane Physical Layer (PHY) specifications and management parameters for operation on backplanes and twinaxial copper cables, and specify optional Energy Efficient Ethernet (EEE) for 40 Gbit/s and 100 Gbit/s operation over backplanes and copper cables.

On May 10, 2013, the P802.3bm 40 Gbit/s and 100 Gbit/s Fiber Optic Task Force was approved.[3]

This project is to specify additions to and appropriate modifications of IEEE Std 802.3 to add 100 Gbit/s Physical Layer (PHY) specifications and management parameters, using a four-lane electrical interface for operation on multimode and single-mode fiber optic cables, and to specify optional Energy Efficient Ethernet (EEE) for 40 Gbit/s and 100 Gbit/s operation over fiber optic cables. In addition, to add 40 Gbit/s Physical Layer (PHY) specifications and management parameters for operation on extended reach (>10 km) single-mode fiber optic cables.

Also on May 10, 2013, the P802.3bq 40GBASE-T Task Force was approved.[12]

Specify a Physical Layer (PHY) for operation at 40 Gbit/s on balanced twisted-pair copper cabling, using existing Media Access Control, and with extensions to the appropriate physical layer management parameters.

On June 12, 2014, the IEEE 802.3bj standard was approved.[2] On February 16, 2015, the IEEE 802.3bm standard was approved.[13]

On May 12, 2016, the IEEE P802.3cd Task Force started working to define next generation two-lane 100 Gbit/s PHY.[14]

On May 14, 2018, the PAR for the IEEE P802.3ck Task Force was approved. The scope of this project is to specify additions to and appropriate modifications of IEEE Std 802.3 to add Physical Layer specifications and Management Parameters for 100 Gb/s, 200 Gb/s, and 400 Gb/s electrical interfaces based on 100 Gb/s signaling. [15]

Early Products[edit]

Optical signal transmission over a nonlinear medium is principally an analog design problem. As such, it has evolved slower than digital circuit lithography (which generally progressed in step with Moore's law). This explains why 10 Gbit/s transport systems existed since the mid-1990s, while the first forays into 100 Gbit/s transmission happened about 15 years later – a 10x speed increase over 15 years is far slower than the 2x speed per 1.5 years typically cited for Moore's law.

Nevertheless, at least five firms (Ciena, Alcatel-Lucent, MRV, ADVA Optical and Huawei) made customer announcements for 100 Gbit/s transport systems[16]   by August 2011 – with varying degrees of capabilities. Although vendors claimed that 100 Gbit/s light paths could use existing analog optical infrastructure, deployment of high-speed technology was tightly controlled and extensive interoperability tests were required before moving them into service.

Designing routers or switches which support 100 Gbit/s interfaces is difficult. The need to process a 100 Gbit/s stream of packets at line rate without reordering within IP/MPLS microflows is one reason for this.

As of 2011, most components in the 100 Gbit/s packet processing path (PHY chips, NPUs, memories) were not readily available off-the-shelf or require extensive qualification and co-design. Another problem is related to the low-output production of 100 Gbit/s optical components, which were also not easily available – especially in pluggable, long-reach or tunable laser flavors.

Backplane[edit]

NetLogic Microsystems announced backplane modules in October 2010.[17]

Multimode fiber[edit]

In 2009, Mellanox[18] and Reflex Photonics[19] announced modules based on the CFP agreement.

Single mode fiber[edit]

Finisar,[20] Sumitomo Electric Industries,[21] and OpNext[22] all demonstrated singlemode 40 or 100 Gbit/s Ethernet modules based on the C Form-factor Pluggable agreement at the European Conference and Exhibition on Optical Communication in 2009.

Compatibility[edit]

Optical fiber IEEE 802.3ba implementations were not compatible with the numerous 40 and 100 Gbit/s line rate transport systems because they had different optical layer and modulation formats as the IEEE 802.3ba Port Types show. In particular, existing 40 Gbit/s transport solutions that used dense wavelength-division multiplexing to pack four 10 Gbit/s signals into one optical medium were not compatible with the IEEE 802.3ba standard, which used either coarse WDM in 1310 nm wavelength region with four 25 Gbit/s or four 10 Gbit/s channels, or parallel optics with four or ten optical fibers per direction.

Test and measurement[edit]

  • Quellan announced a test board in 2009.[23]
  • Ixia developed Physical Coding Sublayer Lanes[24] and demonstrated a working 100GbE link through a test setup at NXTcomm in June 2008.[25] Ixia announced test equipment in November 2008.[26][27]
  • Discovery Semiconductors introduced optoelectronics converters for 100 Gbit/s testing of the 10 km and 40 km Ethernet standards in February 2009.[28]
  • JDS Uniphase introduced test and measurement products for 40 and 100 Gbit/s Ethernet in August 2009.[29]
  • Spirent Communications introduced test and measurement products in September 2009.[30]
  • EXFO demonstrated interoperability in January 2010.[31]
  • Xena Networks demonstrated test equipment at the Technical University of Denmark in January 2011.[32][33]
  • Calnex Solutions introduced 100GbE Synchronous Ethernet synchronisation test equipment in November 2014.[34]
  • Spirent Communications introduced the Attero-100G for 100GbE and 40GbE impairment emulation in April 2015.[35][36]
  • VeEX[37] introduced its CFP-based UX400-100GE and 40GE test and measurement platform in 2012,[38] followed by CFP2, CFP4, QSFP28 and QSFP+ versions in 2015.[39][40]

Mellanox Technologies[edit]

Mellanox Technologies introduced the ConnectX-4 100GbE single and dual port adapter in November 2014.[41] In the same period, Mellanox introduced availability of 100GbE copper and fiber cables.[42] In June 2015, Mellanox introduced the Spectrum 10, 25, 40, 50 and 100GbE switch models.[43]

Aitia[edit]

Aitia International introduced the C-GEP FPGA-based switching platform in February 2013.[44] Aitia also produce 100G/40G Ethernet PCS/PMA+MAC IP cores for FPGA developers and academic researchers.[45]

Arista[edit]

Arista Networks introduced the 7500E switch (with up to 96 100GbE ports) in April 2013.[46] In July 2014, Arista introduced the 7280E switch (the world's first top-of-rack switch with 100G uplink ports).[47]

Extreme Networks[edit]

Extreme Networks introduced a four-port 100GbE module for the BlackDiamond X8 core switch in November 2012.[48]

Dell[edit]

Dell's Force10 switches support 40 Gbit/s interfaces. These 40 Gbit/s fiber-optical interfaces using QSFP+ transceivers can be found on the Z9000 distributed core switches, S4810 and S4820[49] as well as the blade-switches MXL and the IO-Aggregator. The Dell PowerConnect 8100 series switches also offer 40 Gbit/s QSFP+ interfaces.[50]

Chelsio[edit]

Chelsio Communications introduced 40 Gbit/s Ethernet network adapters (based on the fifth generation of its Terminator architecture) in June 2013.[51]

Telesoft Technologies Ltd[edit]

Telesoft Technologies announced the dual 100G PCIe accelerator card, part of the MPAC-IP series.[52] Telesoft also announced the STR 400G (Segmented Traffic Router)[53] and the 100G MCE (Media Converter and Extension).[54]

Commercial trials and deployments[edit]

Unlike the "race to 10 Gbit/s" that was driven by the imminent need to address growth pains of the Internet in the late 1990s, customer interest in 100 Gbit/s technologies was mostly driven by economic factors. The common reasons to adopt the higher speeds were:[55]

  • to reduce the number of optical wavelengths ("lambdas") used and the need to light new fiber
  • to utilize bandwidth more efficiently than 10 Gbit/s link aggregate groups
  • to provide cheaper wholesale, internet peering and data center connectivity
  • to skip the relatively expensive 40 Gbit/s technology and move directly from 10 to 100 Gbit/s

Alcatel-Lucent[edit]

In November 2007, Alcatel-Lucent held the first field trial of 100 Gbit/s optical transmission. Completed over a live, in-service 504 kilometre portion of the Verizon network, it connected the Florida cities of Tampa and Miami.[56]

100GbE interfaces for the 7450 ESS/7750 SR service routing platform were first announced in June 2009, with field trials with Verizon,[57] T-Systems and Portugal Telecom taking place in June–September 2010. In September 2009, Alcatel-Lucent combined the 100G capabilities of its IP routing and optical transport portfolio in an integrated solution called Converged Backbone Transformation.[58]

In June 2011, Alcatel-Lucent introduced a packet processing architecture known as FP3, advertised for 400 Gbit/s rates.[59] Alcatel-Lucent announced the XRS 7950 core router (based on the FP3) in May 2012.[60][61]

Brocade[edit]

Brocade Communications Systems introduced their first 100GbE products (based on the former Foundry Networks MLXe hardware) in September 2010.[62] In June 2011, the new product went live at the AMS-IX traffic exchange point in Amsterdam.[63]

Cisco[edit]

Cisco Systems and Comcast announced their 100GbE trials in June 2008.[64] However, it is doubtful that this transmission could approach 100 Gbit/s speeds when using a 40 Gbit/s per slot CRS-1 platform for packet processing. Cisco's first deployment of 100GbE at AT&T and Comcast took place in April 2011.[65] In the same year, Cisco tested the 100GbE interface between CRS-3 and a new generation of their ASR9K edge router model.[66]

Huawei[edit]

In October 2008, Huawei presented their first 100GbE interface for their NE5000e router.[67] In September 2009, Huawei also demonstrated an end-to-end 100 Gbit/s link.[68] It was mentioned that Huawei's products had the self-developed NPU "Solar 2.0 PFE2A" onboard and was using pluggable optics in CFP form-factor.

In a mid-2010 product brief, the NE5000e linecards were given the commercial name LPUF-100 and credited with using two Solar-2.0 NPUs per 100GbE port in opposite (ingress/egress) configuration.[69] Nevertheless, in October 2010, the company referenced shipments of NE5000e to Russian cell operator "Megafon" as "40GBPS/slot" solution, with "scalability up to" 100 Gbit/s.[70]

In April 2011, Huawei announced that the NE5000e was updated to carry 2x100GbE interfaces per slot using LPU-200 linecards.[71] In a related solution brief, Huawei reported 120 thousand Solar 1.0 integrated circuits shipped to customers, but no Solar 2.0 numbers were given.[72] Following the August 2011 trial in Russia, Huawei reported paying 100 Gbit/s DWDM customers, but no 100GbE shipments on NE5000e.[73]

Juniper[edit]

Juniper Networks announced 100GbE for its T-series routers in June 2009.[74] The 1x100GbE option followed in Nov 2010, when a joint press release with academic backbone network Internet2 marked the first production 100GbE interfaces going live in real network.[75]

In the same year, Juniper demonstrated 100GbE operation between core (T-series) and edge (MX 3D) routers.[76] Juniper, in March 2011, announced first shipments of 100GbE interfaces to a major North American service provider (Verizon[77]).

In April 2011, Juniper deployed a 100GbE system on the UK education network JANET.[78] In July 2011, Juniper announced 100GbE with Australian ISP iiNet on their T1600 routing platform.[79] Juniper started shipping the MPC3E line card for the MX router, a 100GbE CFP MIC, and a 100GbE LR4 CFP optics in March 2012[citation needed]. In Spring 2013, Juniper Networks announced the availability of the MPC4E line card for the MX router that includes 2 100GbE CFP slots and 8 10GbE SFP+ interfaces[citation needed].

In June 2015, Juniper Networks announced the availability of its CFP-100GBASE-ZR module which is a plug & play solution that brings 80 km 100GbE to MX & PTX based networks.[80] The CFP-100GBASE-ZR module uses DP-QPSK modulation and coherent receiver technology with an optimized DSP and FEC implementation. The low-power module can be directly retrofitted into existing CFP sockets on MX and PTX routers.

Standards[edit]

The IEEE 802.3 working group is concerned with the maintenance and extension of the Ethernet data communications standard. Additions to the 802.3 standard[81] are performed by task forces which are designated by one or two letters. For example, the 802.3z task force drafted the original Gigabit Ethernet standard.

802.3ba is the designation given to the higher speed Ethernet task force which completed its work to modify the 802.3 standard to support speeds higher than 10 Gbit/s in 2010.

The speeds chosen by 802.3ba were 40 and 100 Gbit/s to support both end-point and link aggregation needs respectively. This was the first time two different Ethernet speeds were specified in a single standard. The decision to include both speeds came from pressure to support the 40 Gbit/s rate for local server applications and the 100 Gbit/s rate for internet backbones. The standard was announced in July 2007[82] and was ratified on June 17, 2010.[9]

A 40G-SR4 transceiver in the QSFP form factor

The 40/100 Gigabit Ethernet standards encompass a number of different Ethernet physical layer (PHY) specifications. A networking device may support different PHY types by means of pluggable modules. Optical modules are not standardized by any official standards body but are in multi-source agreements (MSAs). One agreement that supports 40 and 100 Gigabit Ethernet is the C Form-factor Pluggable (CFP) MSA[83] which was adopted for distances of 100+ meters. QSFP and CXP connector modules support shorter distances.[84]

The standard supports only full-duplex operation.[85] Other objectives include:

  • Preserve the 802.3 / Ethernet frame format utilizing the 802.3 MAC
  • Preserve minimum and maximum frame size of current 802.3 standard
  • Support a bit error rate (BER) better than or equal to 10−12 at the MAC/PLS service interface
  • Provide appropriate support for OTN
  • Support MAC data rates of 40 and 100 Gbit/s
  • Provide physical layer specifications (PHY) for operation over single-mode optical fiber (SMF), laser optimized multi-mode optical fiber (MMF) OM3 and OM4, copper cable assembly, and backplane.

The following nomenclature is used for the physical layers:[2][3][86]

Physical layer 40 Gigabit Ethernet 100 Gigabit Ethernet
Backplane n.a. 100GBASE-KP4
Improved Backplane 40GBASE-KR4 100GBASE-KR4
7 m over twinax copper cable 40GBASE-CR4 100GBASE-CR10
100GBASE-CR4
30 m over "Cat.8" twisted pair 40GBASE-T
100 m over OM3 MMF 40GBASE-SR4 100GBASE-SR10
100GBASE-SR4
125 m over OM4 MMF[84]
2 km over SMF, serial 40GBASE-FR 100GBASE-CWDM4[87]
10 km over SMF 40GBASE-LR4 100GBASE-LR4
40 km over SMF 40GBASE-ER4 100GBASE-ER4

The 100 m laser optimized multi-mode fiber (OM3) objective was met by parallel ribbon cable with 850 nm wavelength 10GBASE-SR like optics (40GBASE-SR4 and 100GBASE-SR10). The backplane objective with 4 lanes of 10GBASE-KR type PHYs (40GBASE-KR4). The copper cable objective is met with 4 or 10 differential lanes using SFF-8642 and SFF-8436 connectors. The 10 and 40 km 100 Gbit/s objectives with four wavelengths (around 1310 nm) of 25 Gbit/s optics (100GBASE-LR4 and 100GBASE-ER4) and the 10 km 40 Gbit/s objective with four wavelengths (around 1310 nm) of 10 Gbit/s optics (40GBASE-LR4).[88]

In January 2010 another IEEE project authorization started a task force to define a 40 Gbit/s serial single-mode optical fiber standard (40GBASE-FR). This was approved as standard 802.3bg in March 2011.[11] It used 1550 nm optics, had a reach of 2 km and was capable of receiving 1550 nm and 1310 nm wavelengths of light. The capability to receive 1310 nm light allows it to inter-operate with a longer reach 1310 nm PHY should one ever be developed. 1550 nm was chosen as the wavelength for 802.3bg transmission to make it compatible with existing test equipment and infrastructure.[89]

In December 2010, a 10x10 multi-source agreement (10x10 MSA) began to define an optical Physical Medium Dependent (PMD) sublayer and establish compatible sources of low-cost, low-power, pluggable optical transceivers based on 10 optical lanes at 10 Gbit/s each.[90] The 10x10 MSA was intended as a lower cost alternative to 100GBASE-LR4 for applications which do not require a link length longer than 2 km. It was intended for use with standard single mode G.652.C/D type low water peak cable with ten wavelengths ranging from 1523 to 1595 nm. The founding members were Google, Brocade Communications, JDSU and Santur.[91] Other member companies of the 10x10 MSA included MRV, Enablence, Cyoptics, AFOP, oplink, Hitachi Cable America, AMS-IX, EXFO, Huawei, Kotura, Facebook and Effdon when the 2 km specification was announced in March 2011.[92] The 10X10 MSA modules were intended to be the same size as the C Form-factor Pluggable specifications.

On June 12, 2014, the 802.3bj standard was approved. The 802.3bj standard specifies 100 Gbit/s 4x25G PHYs - 100GBASE-KR4, 100GBASE-KP4 and 100GBASE-CR4 - for backplane and twin-ax cable.

On February 16, 2015, the 802.3bm standard was approved. The 802.3bm standard specifies a lower-cost optical 100GBASE-SR4 PHY for MMF and a four-lane chip-to-module and chip-to-chip electrical specification (CAUI-4). The detailed objectives for the 802.3bm project can be found on the 802.3 website.

On May 14, 2018, the 802.3ck project was approved. This has objectives to:[93]

  • Define a single-lane 100 Gb/s Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on 100 Gb/s per lane optical signaling
  • Define a single-lane 100 Gb/s Attachment Unit Interface (AUI) for chip-to-chip applications
  • Define a single-lane 100 Gb/s PHY for operation over electrical backplanes supporting an insertion loss ≤ 28 dB at 26.56 GHz.
  • Define a single-lane 100 Gb/s PHY for operation over twin-axial copper cables with lengths up to at least 2 m.

100G interface types[edit]

Name Clause Media Media
count
Symbol rate
Gigabaud
Symbol coding Notes
100GBASE-KP4 94 (802.3bj)[2] Copper backplane 2⇅ × 4 13.59375 PAM4 × † × RS-FEC(544,514) † 92/90 framing and 31320/31280 lane identification
100GBASE-KR4 93 (802.3bj)[2] 25.78125 NRZ × RS-FEC(528,514)
100GBASE-KR2 137 (802.3cd) 2⇅ × 2 26.5625 PAM4 × RS-FEC(544,514)
100GBASE-CR10 85 (802.3ba)[1] Twin-ax (balanced) copper cable 2⇅ × 10 10.3125 NRZ × 64b66b CXP connector, center 10 out of 12 channels; 7 m reach
100GBASE-CR4 92 (802.3bj)[2] 2⇅ × 4 25.78125 NRZ × RS-FEC(528,514) 5 m reach
100GBASE-CR2 136 (802.3cd) 2⇅ × 2 26.5625 PAM4 × RS-FEC(544,514) specified MDI connectors: QSFP28, microQSFP, QSFP-DD, and OSFP; 3 m reach
100GBASE-SR10 86 (802.3ba)[1] Multi-mode fiber, 850 nm 2⇅ × 10 10.3125 NRZ × 64b66b MPO/MTP connector, center 10 out of 12 channels;
100 m reach on OM3 MMF, 150 m reach on OM4 MMF
100GBASE-SR4 95 (802.3bm)[3] 2⇅ × 4 25.78125 NRZ × RS-FEC(528,514) 70 m reach on OM3 MMF, 100 m reach on OM4 MMF
100GBASE-SR2 138 (802.3cd) 2⇅ × 2 26.5625 PAM4 × RS-FEC(544,514) 100 m reach on OM4 MMF (70 m on OM3)
100GBASE-SR2-BiDi non-IEEE[94] Multi-mode fiber, 850 nm and 900 nm 1↕ × 2 × 2 WDM 26.5625 PAM4 × RS-FEC(544,514) Proprietary. 100m reach on OM4 MMF (70m on OM3).
Each of the two fibers (LC connector) is used for both transmit and receive (on different lambdas).
100GBASE-DR 140 (802.3cd) Single-mode fiber, 1311 nm 2⇅ × 1 53.125 PAM4 × RS-FEC(544,514) 500 m reach
100GBASE-LR4 88 (802.3ba)[1] Single-mode fiber, WDM:

1295.56 nm, 1300.05 nm,
1304.59 nm, 1309.14 nm

2⇅ × 1 × 4 WDM 25.78125 NRZ × 64b66b 10 km reach
100GBASE-ER4 30–40 km reach
100GBASE-CWDM4 MSA non-IEEE[95] Single-mode fiber, WDM:

1271 nm, 1291 nm,
1311 nm, 1331 nm

25.78125 NRZ × RS-FEC(528,514) 2 km reach, multi-vendor non-IEEE Standard.
100GBASE-CWDM4 OCP non-IEEE[96] 500m reach variant derived from CWDM4 to allow cheaper transceivers for datacenter use,
multi-vendor non-IEEE Standard
100GBASE-CLR4 MSA non-IEEE[97] NRZ × RS-FEC(528,514) or
NRZ × 64b66b
2 km reach, multi-vendor non-IEEE Standard.
Interoperable with 100GBASE-CWDM4 when using RS-FEC.
100GBASE-PSM4 non-IEEE[98] Single-mode fiber, 1311 nm 2⇅ × 4 25.78125 NRZ × RS-FEC(528,514) or
NRZ × 64b66b
500m, multi-vendor non-IEEE Standard
100GBASE-ZR non-IEEE[99]

[100]

Single-mode fiber, 1546.119 nm 2⇅ × 1 30.14475 DP-QPSK × SD-FEC 80+ km reach, non-IEEE Standard

Coding schemes[edit]

10.3125 Gbaud with NRZ ("PAM2") and 64b66b on 10 lanes per direction
One of the earliest coding used, this widens the coding scheme used in single lane 10GE and quad lane 40G to use 10 lanes. Due to the low symbol rate, relatively long ranges can be achieved at the cost of using a lot of cabling.
This also allows breakout to 10×10GE, provided that the hardware supports splitting the port.
25.78125 Gbaud with NRZ ("PAM2") and 64b66b on 4 lanes per direction
A sped-up variant of the above, this directly corresponds to 10GE/40GE signalling at 2.5× speed. The higher symbol rate makes links more susceptible to errors.
If the device and transceiver support dual-speed operation, it is possible to reconfigure an 100G port to downspeed to 40G or 4×10G. There is no autonegotiation protocol for this, thus manual configuration is necessary. Similarly, a port can be broken into 4×25G if implemented in the hardware. This is applicable even for CWDM4, if a CWDM demultiplexer and CWDM 25G optics are used appropriately.
25.78125 Gbaud with NRZ ("PAM2") and RS-FEC(528,514) on 4 lanes per direction
To address the higher susceptibility to errors at these symbol rates, an application of Reed–Solomon error correction was defined in IEEE 802.3bj / Clause 91. This replaces the 64b66b encoding with a 256b257b encoding followed by the RS-FEC application, which combines to the exact same overhead as 64b66b. To the optical transceiver or cable, there is no distinction between this and 64b66b; some interface types (e.g. CWDM4) are defined "with or without FEC."
26.5625 Gbaud with PAM4 and RS-FEC(544,514) on 2 lanes per direction
This achieves a further doubling in bandwidth per lane (used to halve the number of lanes) by employing pulse amplitude modulation with 4 distinct analog levels, making each symbol carry 2 bits. To keep up error margins, the FEC overhead is doubled from 2.7% to 5.8%, which explains the slight rise in symbol rate.
53.125 Gbaud with PAM4 and RS-FEC(544,514) on 1 lane per direction
Further pushing silicon limits, this is a double rate variant of the previous, giving full 100GE operation over 1 medium lane.
30.14475 Gbaud with DP-QPSK and SD-FEC on 1 lane per direction
Mirroring OTN4 developments, this employs polarization to carry one axis of the DP-QPSK constellation. Additionally, new soft decision FEC algorithms take additional information on analog signal levels as input to the error correction procedure.
13.59375 Gbaud with PAM4, KP4 specific coding and RS-FEC(544,514) on 4 lanes per direction
A half-speed variant of 26.5625 Gbaud with RS-FEC, with a 31320/31280 step encoding the lane number into the signal, and further 92/90 framing.

40G interface types[edit]

Name Clause Media Media
count
Symbol rate
Gigabaud
Symbol coding Breakout to 4×10G
40GBASE-KR4 84 Copper backplane 2⇅ × 4 10.3125 NRZ × 64b66b Splitter cable / lane separation
40GBASE-CR4 85 Twin-ax (balanced) copper cable
40GBASE-SR4 86 Multi-mode fiber, 850 nm
40GBASE-SR2-BiDi non-IEEE Multi-mode fiber, 850 nm and 900 nm 1↕ × 2 × 2 WDM ? ? not possible
40GBASE-LX4

40GBASE-LM4

non-IEEE Multi-mode or Single-mode fiber, WDM:

1271 nm, 1291 nm,
1311 nm, 1331 nm

2⇅ × 1 × 4 WDM 10.3125 NRZ × 64b66b CWDM (de)multiplexer and CWDM 10G transceivers
40GBASE-LR4 87 Single-mode fiber, WDM:

1271 nm, 1291 nm,
1311 nm, 1331 nm

40GBASE-ER4
40GBASE-PLR4 non-IEEE Single-mode fiber, 1311 nm 2⇅ × 4 Splitter cable / lane separation
40GBASE-FR 89 Single-mode fiber, 1551 nm 2⇅ × 1 41.25 NRZ × 64b66b not possible
40GBASE-T 113 Twisted pair copper cable 1↕ × 4 3.2 PAM16 × (RS-FEC(192,186) + LDPC) not possible (but can autonegotiate to 1×10GBASE-T)
40GBASE-CR4
40GBASE-CR4 ("copper") is a port type for twin-ax copper cable. Its 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its PMD in Clause 85. It uses four lanes of twin-axial cable delivering serialized data at a rate of 10.3125 Gbit/s per lane.[81]
CR4 involves two clauses: CL73 for auto-negotiation, and CL72 for link training. CL73 allows communication between the two PHYs to exchange technical capability pages, and both PHYs come to a common speed and media type. Once CL73 has been completed, CL72 starts. CL72 allows each of the four lanes' transmitters to adjust pre-emphasis via feedback from the link partner.
40GBASE-KR4
40GBASE-KR4 is a port type for backplanes. Normally backplanes are board traces, such as Megtron6 or FR4 materials. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 84. It uses four lanes of backplane delivering serialized data at a rate of 10.3125 Gbit/s per lane.[81]
As in CR4 case, KR4 involves 2 clauses, first CL73 for autoneg, followed by CL72 for link training. CL73 allows the communication between the 2 PHY's to exchange tech ability pages, and both PHYs come to a common speed and media type. Once CL73 has been completed, CL72 starts. CL72 allows each of the 4 lanes transmitter to adjust its preemphasis by way of feedback from the link partner.
40GBASE-SR4
40GBASE-SR4 ("short range") is a port type for multi-mode fiber and uses 850 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 86. It uses four lanes of multi-mode fiber delivering serialized data at a rate of 10.3125 Gbit/s per lane. 40GBASE-SR4 has a reach of 100 m on OM3 and 150m on OM4. There is a longer range variant 40GBASE-eSR4 with a reach of 300 m on OM3 and 400 m on OM4. This extended reach is equivalent to the reach of 10GBASE-SR.[101]
40GBASE-LX4 / 40GBASE-LM4 (naming dependent on vendor)
40GBASE-LX4 / -LM4 adapts the wavelength multiplexing approach known from single-mode fiber for use on multi-mode fiber. The transceivers can generally be used with either single-mode or multi-mode fiber, although they are primarily designed for the latter. When used with multi-mode fiber, they provide up to 140m range on OM3, 160m on OM4[102] while only using a single fiber pair. On single-mode fiber, they operate identically to 40GBASE-LR4, although for some transceivers this mode of operation is out of specification.
40GBASE-LR4
40GBASE-LR4 ("long range") is a port type for single-mode fiber and uses 1300 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 87. It uses four wavelengths delivering serialized data at a rate of 10.3125 Gbit/s per wavelength.[81]
40GBASE-PLR4
40GBASE-PLR4 ("parallel long range") is a variant of 40GBASE-LR4 that uses 4 transmitters at 1310nm over 4 single mode fibers (rather than 4 wavelengths over 1 single mode fiber). Unlike LR4, PLR4 allows breakout to 4x 10G.[103]
40GBASE-ER4
40GBASE-ER4 ("extended range") is a port type for single-mode fiber being defined in P802.3bm and uses 1300 nm lasers. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 87. It uses four wavelengths delivering serialized data at a rate of 10.3125 Gbit/s per wavelength.[3]
40GBASE-FR
40GBASE-FR is a port type for single-mode fiber. Its Physical Coding Sublayer 64b/66b PCS is defined in IEEE 802.3 Clause 82 and its Physical Medium Dependent PMD in Clause 89. It uses 1550 nm optics, has a reach of 2 km and is capable of receiving 1550 nm and 1310 nm wavelengths of light. The capability to receive 1310 nm light allows it to inter-operate with a longer reach 1310 nm PHY should one ever be developed. 1550 nm was chosen as the wavelength transmission to make it compatible with existing test equipment and infrastructure.[81]
40GBASE-T
40GBASE-T is a port type for 4-pair balanced twisted-pair Cat.8 copper cabling up to 30 m defined in IEEE 802.3bq.[104] IEEE 802.3bq-2016 standard was approved by The IEEE-SA Standards Board on June 30, 2016.[105] It uses 16-level PAM signaling over four lanes at 3,200 MBaud each, scaled up from 10GBASE-T.
40GBASE-SR2-BiDi
40GBASE-SR2-BiDi is a proprietary method for transmitting 40G on 2 lambdas over duplex MM fiber. [106] The term BiDi comes from the fact that both fibers are used for both transmit and receive (that is, the fibers are BiDirectional). Its major selling point is its ability to run over existing 10G MM fiber (i.e., it allows users to upgrade from 10G to 40G without upgrading their fiber). With a reach of 100m on OM3 or 150m on OM4 its range is the same as 40GBASE-SR4.

Chip-to-chip/chip-to-module interfaces[edit]

CAUI-10
CAUI-10 is a 100 Gbit/s 10-lane electrical interface defined in 802.3ba.[1]
CAUI-4
CAUI-4 is a 100 Gbit/s 4-lane electrical interface defined in 802.3bm.[3]
100GAUI-4
100GAUI-4 is a 100 Gbit/s 4-lane electrical interface defined in 802.3cd Clause 135D/E.
100GAUI-2
100GAUI-2 is a 100 Gbit/s 2-lane electrical interface defined in 802.3cd Clause 135F/G.

Pluggable Optics Standards[edit]

40G Transceiver Form Factors
The QSFP+ form factor is specified for use with the 40 Gigabit Ethernet. Copper direct attached cable (DAC) or optical modules are supported, see Figure 85–20 in the 802.3 spec. QSFP+ modules at 40Gbit/s can also be used to provide four independent ports of 10 gigabit Ethernet.[1]
100G Transceiver Form Factors
CFP modules use the 10-lane CAUI-10 electrical interface.
CFP2 modules use the 10-lane CAUI-10 electrical interface or the 4-lane CAUI-4 electrical interface.
CFP4 modules use the 4-lane CAUI-4 electrical interface.[107]
QSFP28 modules use the CAUI-4 electrical interface.
SFP-DD or Small Form-factor Pluggable – Double Density modules use the 100GAUI-2 electrical interface.
Cisco's CPAK optical module uses the four lane CEI-28G-VSR electrical interface.[108][109]
There are also CXP and HD module standards.[110] CXP modules use the CAUI-10 electrical interface.

Optical Connectors[edit]

Short reach interfaces use Multiple-Fiber Push-On/Pull-off (MPO) optical connectors; see subclause 86.10.3.3 of the 802.3 spec.[1] 40GBASE-SR4 and 100GBASE-SR4 use MPO-12 while 100GBASE-SR10 uses MPO-24 with one optical lane per fiber strand.

Long reach interfaces use duplex LC connectors with all optical lanes multiplexed with WDM.

See also[edit]

References[edit]

  1. ^ a b c d e f g h "IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force". official web site. IEEE. June 19, 2010. Retrieved June 24, 2011. 
  2. ^ a b c d e f g "100 Gb/s Backplane and Copper Cable Task Force". official web site. IEEE. Archived from the original on 2013-06-27. Retrieved 2013-06-22. 
  3. ^ a b c d e f "40 Gb/s and 100 Gb/s Fiber Optic Task Force". official web site. IEEE. 
  4. ^ "IEEE Forms Higher Speed Study Group to Explore the Next Generation of Ethernet Technology". 2006-07-25. 
  5. ^ "IEEE 802.3 Higher Speed Study Group". IEEE802.org. Retrieved December 17, 2011. 
  6. ^ Jeff Caruso (June 21, 2007). "Group pushes 100 Gigabit Ethernet: The 'Road to 100G' Alliance is born". Network World. Retrieved June 6, 2011. 
  7. ^ "Project Authorization Request Approval notification: Approcal of P802.3ba" (PDF). IEEE Standards Association Standards Board. December 5, 2007. Retrieved June 6, 2011. 
  8. ^ Caruso, Jeff (2008-01-15). "Standardization work on next Ethernet gets under way". NetworkWorld. 
  9. ^ a b "IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force". 2010-06-21. 
  10. ^ "IEEE 802.3ba standard released". Help Net Security web site. June 21, 2010. Retrieved June 24, 2011. The IEEE 802.3ba standard, ratified June 17, 2010, ... 
  11. ^ a b "IEEE P802.3bg 40Gb/s Ethernet: Single-mode Fibre PMD Task Force". official task force web site. IEEE 802. April 12, 2011. Retrieved June 7, 2011. 
  12. ^ "P802.3bq PAR" (PDF). 
  13. ^ "[802.3_100GNGOPTX] FW: P802.3bm-2015 Approval Notification". ieee802.org. Retrieved 2015-02-19. 
  14. ^ "IEEE 802.3 50 Gb/s, 100 Gb/s, and 200 Gb/s Ethernet Task Force". May 12, 2016. 
  15. ^ http://www.ieee802.org/3/ck/P802_3ck_PAR_140518.pdf
  16. ^ "Huawei's 100G is out of the door". 
  17. ^ "NetLogic Microsystems Announces Industry's First Dual-Mode Quad-Port 10GBASE-KR and 40GBASE-KR4 Backplane PHY with Energy Efficient Ethernet". News release. NetLogic Microsystems. October 13, 2010. Retrieved June 24, 2013. 
  18. ^ "Mellanox Technologies". Archived from the original on September 28, 2009. Retrieved September 25, 2009. 
  19. ^ "InterBOARD CFP 100GBASE-SR10 Parallel Optical Module". commercial web site. Reflex Photonics Inc. Archived from the original on September 28, 2009. Retrieved June 7, 2011. 
  20. ^ "Finisar Corporation – Finisar First to Demonstrate 40 Gigabit Ethernet LR4 CFP Transceiver Over 10 km of Optical Fiber at ECOC". Archived from the original on September 28, 2009. Retrieved September 25, 2009. 
  21. ^ "Sumitomo Electric develops 40GbE transceiver". Retrieved September 25, 2009. 
  22. ^ "Hitachi and Opnext unveil receiver for 100GbE and demo 10 km transmission over SMF". Retrieved October 26, 2009. 
  23. ^ "Quellan QLx411GRx 40G Evaluation Board". Archived from the original on September 28, 2009. Retrieved September 25, 2009. 
  24. ^ "Enabling 100 Gigabit Ethernet Implementing PCS Lanes" (PDF). 
  25. ^ "Avago Technologies, Infinera & Ixia to demo the first 100 Gig Ethernet". Archived from the original on 2012-03-10. Retrieved 7 March 2012. 
  26. ^ "Ixia First to Offer 100 GE Testing Capability". News release. Ixia. September 29, 2008. Retrieved June 7, 2011. 
  27. ^ "40 Gb/s and 100 Gb/s Testing: Overview". commercial web site. Ixia. Retrieved June 7, 2011. 
  28. ^ "Discovery Semiconductors – 100 Gb Ethernet (4 x 25 Gb/s) Quad PIN-TIA Optical Receiver". commercial web site. Retrieved June 7, 2011. 
  29. ^ "JDSU Introduces Industry's Most Robust 100 Gigabit Ethernet Test Suite". News release. JDS Uniphase. August 19, 2009. Retrieved June 7, 2011. 
  30. ^ "40/100 GbE: Testing the next generation of high speed Ethernet". commercial web site. Spirent Communications. Retrieved June 7, 2011. 
  31. ^ "EXFO and Opnext Achieve Full Interoperability, Successfully Testing IEEE-Compliant 100 Gigabit Ethernet Optics". News release. January 11, 2010. Retrieved June 7, 2011. 
  32. ^ "Workshop on 100 Gigabit Ethernet a huge success". DTU news. Technical University of Denmark. February 2, 2011. Retrieved June 7, 2011. 
  33. ^ Torben R. Simonsen (January 26, 2011). "Dansk virksomhed klar med test til 100 Gb ethernet". Elektronik Branchen. Retrieved June 7, 2011.  (Danish)
  34. ^ "Calnex Solutions Limited | Calnex Solutions Launches Industry-first 100GbE Tester for Synchronisation". RealWire. Retrieved 2015-10-22. 
  35. ^ "Industry's First 100G Impairment Emulator Helps Reduce the Effect of Latency in High Speed Ethernet Networks". corporate.spirent.com. 15 Apr 2015. Retrieved 2015-10-22. 
  36. ^ "Attero". www.spirent.com. Spirent. Retrieved 15 November 2017. 
  37. ^ "Frost & Sullivan Recognizes VeEX's Technology Development". 
  38. ^ "VeEX introduces industry's smallest multiservice, multitasking analyzer for 40/100G networks. | VeEX Inc. | The Verification EXperts". veexinc.com. Retrieved 2017-02-09. 
  39. ^ "VeEX enhances UX400 Platform with next-gen CFP2 and CFP4 test modules". advanced-television.com. Retrieved 2017-02-09. 
  40. ^ "VeEX unveils 600G testing for its UX400 platform". advanced-television.com. Retrieved 2017-02-09. 
  41. ^ "Mellanox Enables 100Gb/s Interconnect Solution with Introduction of ConnectX-4 Adapter". 
  42. ^ "Mellanox Announces Availability of 100Gb/s Direct Attach Copper and Active Optical Cables". 
  43. ^ "Mellanox Introduces the World's First 25/100 Gigabit Open Ethernet-based Switch". 
  44. ^ Pal Varga (May 1, 2013). "Aitia C-GEP development platform?". FPGA Networking. Retrieved June 6, 2015. 
  45. ^ Pal Varga (June 6, 2016). "FPGA IP core for 100G/40G ethernet?". FPGA Networking. Retrieved June 6, 2016. 
  46. ^ Jim Duffy (May 1, 2013). "Arista heading off Cisco/Insieme at 100G SDNs?". Network World. Retrieved May 24, 2013. 
  47. ^ Kristin Bent (July 16, 2014). "Arista Leading 100GbE Charge With 7280E Switch Series Launch". CRN. Retrieved February 18, 2016. 
  48. ^ Duffy, Jim (November 13, 2012). "Extreme joins Cisco, Brocade, Huawei at 100G". Network World. p. 1. Retrieved January 18, 2013. 
  49. ^ Dell Force10 S-series model comparison, visited 2 March 2013
  50. ^ Technical details PowerConnect 8100 series, visited: 2 March 2013
  51. ^ "Chelsio Delivers 40Gb Ethernet Adapter (40GbE), Sets new performance bar for high speed Ethernet". Press release. June 11, 2013. Archived from the original on 2013-06-22. Retrieved June 20, 2013. 
  52. ^ http://telesoft-technologies.com/technologies/mpac-ip-7200-dual-100g-ethernet-accelerator-card[dead link]
  53. ^ http://telesoft-technologies.com/technologies/str-400g[dead link]
  54. ^ http://telesoft-technologies.com/technologies/100g-mce-media-converter-and-extension[dead link]
  55. ^ 100G in routers Juniper Networks Presentation at ECOC 2009
  56. ^ "Verizon Successfully Completes Industry's First Field Trial of 100 Gbps Optical Network Transmission". 
  57. ^ "Verizon completes industry-leading 100G Ethernet field trial". 
  58. ^ "A game-changing approach to the core". 
  59. ^ "Alcatel-Lucent's FP3 network processor routes at 400Gbps". Press release. June 29, 2011. Retrieved June 24, 2013. 
  60. ^ David Goldman (May 21, 2012). "How Alcatel-Lucent made the Internet 5 times faster". CNN Money. Retrieved June 24, 2013. 
  61. ^ "100 Gigabit Ethernet (100GE): Services unleashed at speed". Company web site. Archived from the original on November 16, 2012. Retrieved June 24, 2013. 
  62. ^ Brocade set to unveil 100G Ethernet Brocade
  63. ^ "3 new services are launched by AMS-IX at MORE IP event". 
  64. ^ "Cisco NGN Transport Solutions" (PDF). 
  65. ^ Matsumoto, Craig (April 11, 2011). "AT&T, Comcast Go Live With 100G". Light Reading. Retrieved December 17, 2011. 
  66. ^ Liu, Stephen (July 25, 2011). "Cisco Live! Showing Off 100GbE on CRS-3 and ASR 9000 Series". blogs.cisco.com. Retrieved December 17, 2011. 
  67. ^ "Huawei Successfully Develops 100 Gigabit Ethernet WDM Prototype". 
  68. ^ "Huawei Launches World' s First End-to-End 100G Solutions". 
  69. ^ "Huawei E2E 100G Solution" (PDF). 
  70. ^ "Russia's MegaFon Awards Backbone Contract to Huawei". 
  71. ^ "Huawei Unveils the World's First 200G High-Speed Line Card for Routers". 
  72. ^ "Huawei 200G Solution". 
  73. ^ "Оборудование Huawei 100G успешно прошло тестирование в России". 
  74. ^ "Juniper networks introduces breakthrough 100 gigabit Ethernet interface for t series routers". 
  75. ^ "Internet2 racing ahead with 100G Ethernet network". 
  76. ^ "Juniper Demonstrates Industry's First Live 100G Traffic From the Network Core to Edge". 
  77. ^ "Verizon First Service Provider to Announce 100G Deployment on U.S. Network". 
  78. ^ Deploying 100GE JANET UK
  79. ^ "iiNet Pioneers 100GbE with new Juniper Networks Backbone". 
  80. ^ "Juniper networks - Life Begins at 40(km) - 100G ZR Optics". 
  81. ^ a b c d e "IEEE 802.3 standard". 
  82. ^ Reimer, Jeremy (2007-07-24). "New Ethernet standard: not 40Gbps, not 100, but both". ars technica. 
  83. ^ "CFP Multi-Source Agreement". official web site. Archived from the original on September 28, 2009. Retrieved June 24, 2011. 
  84. ^ a b Greg Hankins (October 20, 2009). "IEEE P802.3ba 40 GbE and 100 GbE Standards Update" (PDF). North American Network Operators' Group (NANOG) 47 Presentations. Retrieved June 24, 2011. 
  85. ^ John D'Ambrosia. "IEEE P802.3ba Objectives" (PDF). Archived (PDF) from the original on September 28, 2009. Retrieved September 25, 2009. 
  86. ^ Ilango Ganga (May 13, 2009). "Chief Editor's Report" (PDF). IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force public record. p. 8. Retrieved June 7, 2011. 
  87. ^ "CWDM4-MSA Group". Retrieved 2015-07-24. 
  88. ^ Ilango Ganga; Brad Booth; Howard Frazier; Shimon Muller; Gary Nicholl (May 13, 2008). "IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force, May 2008 Meeting". 
  89. ^ Anderson, Jon. "Rationale for dual-band Rx in 40GBASE-FR" (PDF). 
  90. ^ "10 x 10 MSA – Low Cost 100 GB/s Pluggable Optical Transceiver". official web site. 10x10 multi-source agreement. Retrieved June 24, 2011. 
  91. ^ "Leading Industry Peers Join Forces to Develop Low-Cost 100G Multi-Source Agreement". Businesswire news release. December 7, 2010. Retrieved June 24, 2011. 
  92. ^ "10X10 MSA Ratifies Specification for Low Cost 100 Gb/s 2 Kilometer Links" (PDF). News release. 10x10 MSA. March 4, 2011. Retrieved June 24, 2011. 
  93. ^ http://www.ieee802.org/3/ck/P802_3ck_Objectives_2018mar.pdf
  94. ^ "Looks Like We're Upgrading Again! Dual-Rate 40G/100G BiDi Transceiver". Cisco. Retrieved 23 April 2018. 
  95. ^ "CWDM4-MSA Group". 
  96. ^ "100GBase-CWDM4 OCP standards". 
  97. ^ "CLR4-Alliance Group". 
  98. ^ "100G PSM4 MSA". 
  99. ^ "OIF coherent optics agreements". 
  100. ^ "100GBASE-ZR". 
  101. ^ Coleman, Doug. "Optical Trends in the Data Center" (PDF). 
  102. ^ "40G Ethernet LM4 140m Duplex Multimode QSFP+ Optical Transceiver". 
  103. ^ Li, Angelina. "Comparison between QSFP-40G-LR4 and QSFP-40G-PLR4". linkedin.com. Retrieved 5 June 2017. 
  104. ^ "IEEE P802.3bq 40GBASE-T Task Force". IEEE 802.3. 
  105. ^ "Approval of IEEE Std 802.3by-2016, IEEE Std 802.3bq-2016, IEEE Std 802.3bp-2016 and IEEE Std 802.3br-2016". IEEE. 2016-06-30. .
  106. ^ "Migrate to a 40-Gbps Data Center with Cisco QSFP BiDi Technology". Cisco. Retrieved 23 April 2018. 
  107. ^ "CFP MSA". 
  108. ^ "Cisco CPAK 100GBASE Modules Data Sheet". 
  109. ^ "Multi-Vendor Interoperability Testing of CFP2, CPAK and QSFP28 with CEI-28G-VSR and CEI-25G-LR Interface During ECOC 2013 Exhibition" (PDF). 
  110. ^ Daniel Dove. "4X25G Optical Modules and Future Optics" (PDF). Archived (PDF) from the original on 2013-07-12. Retrieved 2013-07-04. 

Further reading[edit]

External links[edit]