Thread Links Date Links
Thread Prev Thread Next Thread Index Date Prev Date Next Date Index

Re: [802.3BA] 40G/100G SMF distinct identity [WAS: RE: [802.3BA] 5 Criteria mod to support 40 G on SMF]

I agree that there are many differences between Ethernet and Fibre Channel protocols and markets.  Both protocols have shared certain aspect at the physical layers and that is where I think it is helpful to compare and contrast the two. 
Any system or organism that grows exponentially by doubling will reach limitations.  I did not try to create a rule that FC will continue to double indefinitely without increase in cost.  I was trying to report what happened, not predict the future. 
The cost of 8GFC is less than one half of the cost of 1GFC in 2001 without even considering inflation.  This is mainly due to the lower volumes of 1GFC when prices were relatively high.  1GFC ports were shipped in 16 or 32 port configurations while  8GFC products are now shipping with 768 port backbone switches to a much larger market in higher volumes.
Ethernet does typically span larger topologies than FC, but this is not a limit of the Fibre Channel architecture.  Fibre Channel has been successfully mapped to SONET and ATM (Fibre Channel Backbone - FC-BB), Internet Protocol (FC-BB-2 and RFC 3821), Generic Framing Protocol (FC-BB-3) and Pseudo Wire (FC-BB-4).  Fibre Channel is even now being mapped to Ethernet in FC-BB-5.  As long as response times (round trip latency) are below Error Detect Time Out Values of 2 seconds, most applications will work fine on earth.  The press and pundits often don't realize that the Fibre Channel protocols are not limited in distance, only in the demands that customers place upon it.
I'm not suggesting that Ethernet should have gone to 2G, but I bet users would have adopted it if it was backward compatible. 
FC and Ethernet address different markets and needs as you say.  I agree that 10GE was not designed for laptops or even PCs or servers.  If 10GE was designed for servers, it would have been designed for a shorter distance than 300 meters.  The 300 meter requirement drove the expensive lasers, tight jitter budgets and difficult testing requirements.  I have quoted Finisar's presentation on this ( ) a number of times and most transceiver companies agree that 10GE is unnecessarily difficult to manufacture.  I have heard that the 10GE transceiver costs could be reduced by 40% if the distance was decreased to 100 meters. 
8GFC PHY was designed to be lower cost and backward compatible to earlier speeds.  8GFC links only spans 150 meters on OM3 fiber (still longer than 40/100GE) and has better allocation of jitter budgets than 10GE where too much jitter budget was allocated to fiber at the detriment of the electronics.  Proper definition of 8GFC for a different market between servers and storage has kept the cost in 8GFC physical layer costs considerably lower than 10GE.
I know many users would like to move to 10GE for their server interconnects since they need to use multiple 1GE links now.  With multicore processors and blade server chassis, you'd think that more than 100,000 10GE ports would have shipped on the 7+ million servers shipped in 2007
If some server vendors could comment on why there is low adoption of 10GE to the server, I'd like to know why if it isn't cost.

From: sanjeev mahalawat [mailto:sanjeevmahalawat@GMAIL.COM]
Sent: Monday, March 03, 2008 12:23 PM
Subject: Re: [802.3BA] 40G/100G SMF distinct identity [WAS: RE: [802.3BA] 5 Criteria mod to support 40 G on SMF]

Agree that cost plays a big role. However, if 2x increments keep the cost same (or lower)---as you mentioned---then why is 8GFC more expansive than 1G (rule of transitive; 2G<=1G, 4G<=2G, 8G<=4G)?
I think Ethernet networks span larger topologies than FC (which is constrained and limited to and by computing/storage topologies). One can't compare the two directly.
I think 10GE didn't take off in the HBA not because of cost of Ethernet, but the cost of filling up the 10GE port and perhaps, not being able to fill that pipe. Since that did not happen the volime did not happen and when that did not happen cost did not come down as fast.
Just my opinion, I don't think 10GE was envisioned to be in laptop machines and not sure if there was 2G Ethernet even designed to be compatible with 10/100 how would have added any value to these machines ---which are ading to the volume---with 1GE link?
Also, if I recall correctly 1G FC was very expansive until 1G Ethernet took off.
My $0.0.

On 3/1/08, Scott Kipp <> wrote:

You said:

Being a data communications guy, I don't understand why FC seems to be
increasing by factor 2 each generation, instead of factor 4 or 10.
Representatives for Brocade (in my mind primarily FC vendor) says
customers want small increases in link speed, which is not my

I'm the Brocade representative, so I'll attempt to answer the question
regarding why FC continues to grow by factors of 2 instead of 4 or 10.
I never said customers want small increases in link speed, but I'll say
that they want it if its no additional cost or even a reduction in cost.

Fibre Channel has always attempted to be the low cost leader in fiber
optic bandwidth. Fibre Channel standardized the first 1 gigabit links
based on VCSELs that were very successful.  Ethernet saw the advantage
of this and soon copied large parts of the specification and developed
highly successful 1 Gig fiber optic links using the same technologies.
Most 1 Gig FC and 1 Gig Ethernet fiber optic transceivers work on either

Many small improvements in transceivers lead to big differences.  Fibre
Channel saw that they could soon advance to 2 Gig and be backward
compatible with 1 gig for basically no additional cost or even less cost
when new processes were developed.   Customers liked doubling the speed
for no additional cost and it still autonegotiated down to 1 Gig if
needed. The same shift was seen for 4 Gig when almost all 2 Gig gear was
obsoleted by the new gear.  The speed doubling usually coincided with
new ASIC development and gained other capabilities with the speed
increase. Speed sells and there were no interoperability problems
between the speeds.

You see much smaller speed jumps in microprocessor speeds and disk drive
capacities.  Does anyone complain when a processor increases from 2.1
GHz to 2.4 GHz as long as it works?

Fibre Channel is currently going through the transition to 8 Gig and
there is currently a cost premium over 4 Gig solutions, so the
transition to 8 Gig might not be as fast as from 1GFC to 2GFC or 2GFC to
4GFC, but the cost difference is diminishing and customers are adopting
8GFC rapidly in the first quarter of released products.

Fibre Channel did make the mistake of following Ethernet into the 10 Gig
space.  10GFC has seen minimal adoption because of the high cost premium
over even 8GFC. No servers have adopted 10GFC and FC HBAs are not
expected to move to 10GFC but to 8GFC and then 16GFC.  Ethernet has seen
a very similar lack of adoption of 10 Gig because it isn't backward
compatible with 1GE.  GigE has been very successful because of its
backward compatibility with 10/100.  In 2007, over 1 million 10GE ports
were shipped compared to over 100M Gig E ports.  Six years after the 10
Gig standard was complete, less than 1% of Ethernet ports are 10 Gig.

I was at OFC this week and the high cost of 10 Gig was a big topic of
debate. We're still in a chicken and egg scenario with 10 Gig.  Volumes
could drive the cost of 10 Gig down, but we haven't seen it yet.  Users
would like 10 Gig on servers, but mainly buy 1 Gig.  I've heard that
only about 100,000 servers connected with 10GE last year.

I've tried to make the point that users don't mind minimal speed
increases for minimal or no cost increases as long as they are backward
compatible.  Fibre Channel is attempting to continue that tradition with
8GFC and 16GFC.  Just in time technologies that have been optimized
instead of before there time technologies that are not cost-effective.

My 10 cents,
Scott Kipp

-----Original Message-----
From: Mikael Abrahamsson [mailto:swmike@SWM.PP.SE]
Sent: Saturday, February 16, 2008 12:35 PM
Subject: Re: [802.3BA] 40G/100G SMF distinct identity [WAS: RE:
[802.3BA] 5 Criteria mod to support 40 G on SMF]

On Sat, 16 Feb 2008, Alessandro Barbieri (abarbier) wrote:

> This group has identified a minimum set of applications that is
> sufficient to justify the DI of 40G and 100G, that should be enough.
> We can't pretend 40G will not find its way into the aggregation. Much
> like in 10-12 years 100G will find its way into the server market.

Sorry for my previous email where I used "price" instead of "cost". I'll
try to improve.

So, backing up a bit, I'm trying to see this from a system perspective,
ie in the equipment where these technologies are going to be used.

I have some thinking I would like some vendors to give feedback on,
which I think is important in this decision as well.

Currently, a lot of the current highend platforms use 40 gigabit/s
backplane connectivity. Would they get 40GE support on the current
chassis/fabric generation in 2010?

By 2010, my guess is that highend platforms will have backplane
connectivity somewhere in the 100, 120, 160 or 200 gigabit/s speeds.

If 100, this would enable single port 100GE or two port 40GE blades.
If 120, this would enable single port 100GE or three port 40GE blades.
If 160, this would enable single port 100GE or four port 40GE blades.
If 200, this would enable dual port 100GE or five port 40GE blades.

Would the 40GE blades have pluggable optics that would support all
proposed reaches? If there is a push for 10km SMF reach for 40GE, why
not 40 or 80km reach there as well? With 3 or 4 port 40GE blades one
would have capacity to on a single blade create rings and then use 40GE
as aggregation technology immediately. Then it makes sense to have 10
and 40km SMF reaches (and seeing that 10GBASE-ZR is actually used quite
a bit, wouldn't 80km as well be of interest to standardize by the same
logic that vendors will standardize 40GE-10km SMF if IEEE doesn't)?

I also think that any proponent for 40GE needs to explain what the next
step after 100GE should be, as 40GE creates a precedent for 250GE or
400GE as next step if the logic is to be intact.

Being a data communications guy, I don't understand why FC seems to be
increasing by factor 2 each generation, instead of factor 4 or 10.
Representatives for Brocade (in my mind primarily FC vendor) says
customers want small increases in link speed, which is not my
In the ISP world people often want to use their equipment for a long
time and write them off in 3-6 years, as they are usually quite big and
big in capital spending, thus the need for factor 4 or 10 in speed
increase over the previous generation.

I also feel that 40GE was pushed as server technology needed in 2012, in
order to make 40GE worthwile it needs to be here before 100GE, not
after, or show big cost savings so as to be more economical per bit
compared to 100GE. Being compatible with existing equipment (new
linecard for existing
platforms) might cause it to be more economical for end users from a
total expenditure perspective.

Being an ISP guy, I also feel that any push for 40GE needs to have broad
switch/router vendor support, because if the core routing platforms
doesn't support 40GE then there is a big hurdle in how to connect the
aggregation to the core for any larger sized ISP who has a WAN.

But then again, if 40GE 10km SMF uses a lot of 10GE technology and has a
2.5x cost increase over 10GE, and 100GE has 10x cost increase, and 40GE
can be brought quickly to market and 100GE will take more r&d and is
harder to realise, then 40GE SMF might still have a place.

But I do have to say again: 40GE cannot delay 100GE effort, it's needed
soon of not NOW. 40GE is bandaid for ISPs, the solution is 100GE if not
higher speeds, or we risk stalling the growth of the Internet.

Mikael Abrahamsson    email: