Re: Wide Area Networking for the Rest of US - the debate on BER and other issues

[pbottorf@nortelnetworks.com (Paul Bottorff)]

Larry:

When the entire link bandwidth is being used for a single file transfer and
the transmit window used by higher layers is near the BER then the system
will have poor performance. The issue is whether this is a realistic
application on a 10 GigE network. For a computer room application where
transfers are bulk file transfers using upper layer stream oriented
protocols the required transmission window is reduced, since the latency to
the next rack is small, minimizing the impact of the BER. For applications
like wide area backbones where the link carries an aggregation of many slow
speed flows any given flow will not see many errors.

A more precise way to look at this issue is that the BER should be much
less than the bandwidth*delay product for any flow. For a 10 G line at 100
meters the bandwidth*delay product is about: 10 Gbit/sec * 500 nsec = 5000
bits. For 1000K meters the bandwidth*delay product is about 50,000,000
bits. Both are way under 10^12 bits. 

Paul

At 08:37 AM 6/2/99 -0700, Larry Miller wrote:
>
>I think the bit is that when you report bad frames upward to higher layers
>they have to do some work to re-request those frames and that takes much
>longer than the time actually burned by the dropped frames. Hence, if you
>get too low of a raw BER you spend all (or maybe more than all) of your time
>with higher layer thrashing and never get through with the (say) file
>transfer.
>
>This, I think, is the fallacy in Mr St. Arnaud's notion.
>
>Larry Miller
>Nortel Networks
>
>
>-----Original Message-----
>From: Mike Dudek <mdudek@cieloinc.com>
>To: Chang, Edward S <Edward.Chang@unisys.com>
>Cc: bin.guo@amd.com <bin.guo@amd.com>; bill.st.arnaud@canarie.ca
><bill.st.arnaud@canarie.ca>; rtaborek@transcendata.com
><rtaborek@transcendata.com>; dwmartin@nortelnetworks.com
><dwmartin@nortelnetworks.com>; stds-802-3-hssg@ieee.org
><stds-802-3-hssg@ieee.org>; sachs@watson.ibm.com <sachs@watson.ibm.com>
>Date: Tuesday, June 01, 1999 5:42 PM
>Subject: Re: Wide Area Networking for the Rest of US - the debate on BER and
>other issues
>
>
>>
>>Agreed, but the percentage of good frames stays the same.  ie the
>percentage
>>bandwidth used for retransmissions is the same.
>>
>>"Chang, Edward S" wrote:
>>
>>> Mike:
>>>
>>> If the BER is maintained the same for both GbE and 10xGbE and assume
>>> everything is equal, the frequency of getting error from 10GbE is 10
>times
>>> than GbE from PHY.  Of course, the whole system has other factors to be
>>> included to find the final throughput.  In another word, the occurrence
>of
>>> frame error will be much more for 10GbE than GbE.
>>>
>>> I may present mathematical analysis in July, if my time is allowed.
>>>
>>> Ed Chang
>>> Unisys Corporation
>>>
>>> -----Original Message-----
>>> From: Mike Dudek [mailto:mdudek@cieloinc.com]
>>> Sent: Tuesday, June 01, 1999 10:07 AM
>>> To: Chang, Edward S
>>> Cc: bin.guo@amd.com; bill.st.arnaud@canarie.ca;
>>> rtaborek@transcendata.com; dwmartin@nortelnetworks.com;
>>> stds-802-3-hssg@ieee.org; sachs@watson.ibm.com
>>> Subject: Re: Wide Area Networking for the Rest of US - the debate on BER
>>> and other issues
>>>
>>> I do not agree that the BER must be improved with data rate increase in
>>> order to
>>> obtain the higher throughput.  At least for packet based transmission
>with
>>> retransmission of errored packets, the throughput increases in proportion
>to
>>> the
>>> data rate for the same BER, assuming that the packet length (in bytes)
>>> remains
>>> fixed.  I do not think that anyone has proposed changing the packet
>length,
>>> but
>>> if they did then the BER might have to be improved.  The throughput is of
>>> course
>>> the number of good packets in any interval of time.
>>>
>>> "Chang, Edward S" wrote:
>>>
>>> > Bin:
>>> >
>>> > Yes, I agree.  The BER should be improved with data rate increase, if
>the
>>> > through put gained from higher data rate is to be maintained.  In
>addition
>>> > to the retry times wasted, the external sources of noise remain the
>same,
>>> > which further requires the lower BER.  These are the correct design
>goals
>>> we
>>> > should work on.  Although, we also should keep the cost-effectiveness
>in
>>> > mind to maintain optimum balance between performance and cost.
>>> >
>>> > Ed Chang
>>> > Unisys Corporation
>>> >
>>> > -----Original Message-----
>>> > From: bin.guo@amd.com [mailto:bin.guo@amd.com]
>>> > Sent: Friday, May 28, 1999 4:57 PM
>>> > To: Edward.Chang@unisys.com; bill.st.arnaud@canarie.ca;
>>> > rtaborek@transcendata.com; dwmartin@nortelnetworks.com
>>> > Cc: stds-802-3-hssg@ieee.org; sachs@watson.ibm.com; "widmer@us.ibm.com
>>> > widmer@us.ibm.com widmer"@us.ibm.com
>>> > Subject: RE: Wide Area Networking for the Rest of US - the debate on
>BER
>>> > a nd other issues
>>> >
>>> > Ed,
>>> >
>>> > If the specified BER for 1000BASE-X is 10^ -12, then to have the equal
>>> > error-free period the specified BER for 10G should be at least 10^ -13.
>>> > Based on Rich T and Rich S's BER number:
>>> >
>>> > A system BER of 10 E - 8 @  10 Mbps = a bit error every 10 seconds.
>>> > (10BASE-T)
>>> > A system BER of 10 E-12 @ 100 Mbps = a bit error every 166 minutes, 40
>>> > seconds.        (100BASE-X)
>>> > A system BER of 10 E-10 @     1 Gbps = a bit error every 1 minutes, 40
>>> > seconds.                (1000BASE-T)
>>> > A system BER of 10 E-12 @     1 Gbps = a bit error every 16 minutes, 40
>>> > seconds.        (1000BASE-X)
>>> > A system BER of 10 E-12 @   10 Gbps = a bit error every 1 minutes, 40
>>> > seconds.
>>> > A system BER of 10 E-13 @   10 Gbps = a bit error every 16 minutes, 40
>>> > seconds.
>>> >
>>> > If the TCP/IP is the only protocol 10G PHY needs to support, then the
>>> above
>>> > specified BER may be more than enough.  Moving from 1G to 10G, the bit
>>> > period is scaled 10X smaller while jitter and noise from some sources
>are
>>> > not scaled the same way -- much tight control should be applied to
>achieve
>>> > even the same BER.
>>> >
>>> > Bin
>>> >
>>> > ADL,AMD
>>> >
>>> >
>>> >
>>> >
>>> > > -----Original Message-----
>>> > > From: Chang, Edward S [SMTP:Edward.Chang@unisys.com]
>>> > > Sent: Friday, May 28, 1999 12:44 PM
>>> > > To:   bill.st.arnaud@canarie.ca; Guo, Bin; rtaborek@transcendata.com;
>>> > > dwmartin@nortelnetworks.com
>>> > > Cc:   stds-802-3-hssg@ieee.org; sachs@watson.ibm.com;
>"widmer@us.ibm.com
>>> > > widmer@us.ibm.com          widmer"@us.ibm.com
>>> > > Subject:      RE: Wide Area Networking for the Rest of US - the
>debate
>>> on
>>> > > BER a nd other issues
>>> > >
>>> > > Bill:
>>> > >
>>> > > I like your idea of implementing native 10xGBE for intermediate long
>>> haul
>>> > > and WAN, which is a good move.  The advantage you are mentioning will
>>> > > greatly reduce the cost to users.
>>> > >
>>> > > It is true, in a TCP/IP links, the TCP flow control causes more
>>> > > retransmission than BER. Therefore, the extremely low BER, 10^-15,
>does
>>> > > not
>>> > > necessarily gain any more advantage than the specified BER of 10^-12.
>>> > >
>>> > >
>>> > > Ed Chang
>>> > >
>>> > > -----Original Message-----
>>> > > From: Bill St. Arnaud [mailto:bill.st.arnaud@canarie.ca]
>>> > > Sent: Friday, May 28, 1999 8:52 AM
>>> > > To: bin.guo@amd.com; rtaborek@transcendata.com;
>>> > > dwmartin@nortelnetworks.com
>>> > > Cc: stds-802-3-hssg@ieee.org; sachs@watson.ibm.com;
>"widmer@us.ibm.com
>>> > > widmer@us.ibm.com widmer"@us.ibm.com
>>> > > Subject: Wide Area Networking for the Rest of US - the debate on BER
>and
>>> > > other issues
>>> > >
>>> > >
>>> > >
>>> > > All:
>>> > > I have been following the interesting debate about BER. Let me bring
>>> some
>>> > > further issues into the debate.
>>> > >
>>> > > I am assuming that on WAN and long haul GbE the upper layer protocol
>>> will
>>> > > only be IP.
>>> > >
>>> > > On most IP links, even ones with BERs of 10^-15 there is about 1-3%
>>> packet
>>> > > loss and retransmission.  This is due to a number of factors but most
>>> > > typically it relates to TCP flow control mechanism from server bound
>>> > > congestion (not network congestion) and the use of WRED in routers.
>>> > >
>>> > > So, on most IP links the packet loss due to BER is significantly less
>>> than
>>> > > that due to normal TCP congestion.  As long as that ratio is
>maintained
>>> it
>>> > > is largely irrelevant what the absolute BER value is.  There will be
>>> many
>>> > > more retransmissions from the IP layer than there will be at the
>>> physical
>>> > > layer due to BER.
>>> > >
>>> > > Other protocols like Frame Relay and SNA are a lot more sensitive to
>>> high
>>> > > BERs.  IP ( in particular TCP/IP) is significantly more robust and
>can
>>> > > work
>>> > > quite effectively in high BER environments e.g. TCP/IP over barbed
>wire.
>>> > >
>>> > > I would like to suggest that the 802.3 HSSG group consider an 2
>>> solutions
>>> > > for 10xGbE WAN:
>>> > > (1) native 10xGbE using 8b/10b; and
>>> > > (2)10xGbE mapped to a SONET STS OC-192 frame
>>> > >
>>> > > For extreme long haul solutions SONET makes a lot of sense as a
>>> transport
>>> > > technology.  However for intermediate long haul (up to 1000 km) and
>WAN
>>> > > native 10xGbE is more attractive. Native GbE can be either
>transported
>>> on
>>> > > a
>>> > > transparent optical network or carried directly on a CWDM system with
>>> > > transceivers. In medium range networks coding efficiency is not as
>>> > > important
>>> > > as it is in long haul networks. If coding efficiency is important
>then
>>> in
>>> > > my
>>> > > opinion, it does not make sense to invent a new coding scheme for
>10xGbE
>>> > > when it would be just as easy to map it to a SONET frame.
>>> > >
>>> > > The attraction of native 10xGbE for the WAN is that it is a "wide
>area
>>> > > networking solution for the rest of us".  You don't need to hire
>>> > > specialized
>>> > > SONET engineers to run and manage your networks.  The 18 year old kid
>>> who
>>> > > is
>>> > > running your LAN can now easily learn to operate and manage a WAN.
>>> > >
>>> > > In Canada and the US, there are several vendors who are willing to
>sell
>>> > > dark
>>> > > fiber at a very reasonable cost.  Right now the cost of building a
>WAN
>>> > > with
>>> > > 10xGbE and CWDM is substantially less (for comparable data rates)
>than
>>> > > using
>>> > > SONET equipment.
>>> > >
>>> > > Bill
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > -------------------------------------------
>>> > > Bill St Arnaud
>>> > > Director Network Projects
>>> > > CANARIE
>>> > > bill.st.arnaud@canarie.ca
>>> > > http://tweetie.canarie.ca/~bstarn
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > > -----Original Message-----
>>> > > > From: owner-stds-802-3-hssg@majordomo.ieee.org
>>> > > > [mailto:owner-stds-802-3-hssg@majordomo.ieee.org]On Behalf Of
>>> > > > bin.guo@amd.com
>>> > > > Sent: Thursday, May 27, 1999 7:28 PM
>>> > > > To: rtaborek@transcendata.com; dwmartin@nortelnetworks.com
>>> > > > Cc: stds-802-3-hssg@ieee.org; sachs@watson.ibm.com;
>"widmer@us.ibm.com
>>> > > > widmer@us.ibm.com widmer"@us.ibm.com
>>> > > > Subject: RE: 1000BASE-T PCS question
>>> > > >
>>> > > >
>>> > > >
>>> > > > Rich,
>>> > > >
>>> > > > The DC balance can be directly translated into jitter (when timing
>is
>>> > > > concerned) and offset (when threshold slicing is concerned).  You
>>> > > > only need
>>> > > > to deal with the former if the signal is 2-level NRZI, while you
>need
>>> to
>>> > > > deal with both if multi-level signal modulation is used.
>>> > > >
>>> > > > For long term DC imbalance, it translates into low frequency jitter
>>> and
>>> > > if
>>> > > > it's low enough(<1 KHz ?), it's called baseline wonder.  For
>>> > > > short term, it
>>> > > > relates to Data Dependent Jitter, which is more difficult for
>timing
>>> > > > recovery to handle since it's not from system or channel imparity,
>and
>>> > > > therefore it's harder to compensate.
>>> > > >
>>> > > > When you have a lot of jitter margin, for example in lower speed
>>> > > clocking,
>>> > > > the amount of jitter, translated from DC drift resulted from data
>>> > > > imbalance
>>> > > > coupled by AC circuit, percentage wise is a small portion of the
>clock
>>> > > > period and therefore does not contribute to much of the eye
>>> > > > closing.  On the
>>> > > > other hand, for high speed clocking at 10G (100 ps?), the jitter
>>> > > > translated
>>> > > > from the same amount of DC drift can be a significant portion of
>the
>>> > > clock
>>> > > > period, so contributes to much large percentage wise jitter which
>>> > > > results in
>>> > > > reduced eye opening -- higher BER.
>>> > > >
>>> > > > Dave said in his mail that "The limiting factor is enough RX
>optical
>>> > > power
>>> > > > to provide a sufficiently open eye." but you still have to deal
>with
>>> the
>>> > > > data dependent jitter due to DC imbalance generated after O/E, that
>>> can
>>> > > > close the eye further again.
>>> > > >
>>> > > > Bin
>>> > > >
>>> > > > ADL, AMD
>>> > > >
>>> > > > > -----Original Message-----
>>> > > > > From:     Rich Taborek [SMTP:rtaborek@transcendata.com]
>>> > > > > Sent:     Thursday, May 27, 1999 3:23 PM
>>> > > > > To:       David Martin
>>> > > > > Cc:       HSSG_reflector; Sachs,Marty; Widmer,Albert_X
>>> > > > > Subject:  Re: 1000BASE-T PCS question
>>> > > > >
>>> > > > >
>>> > > > > Dave,
>>> > > > >
>>> > > > > Do you know of any research or other proofs in this area? You say
>>> that
>>> > > > > lower speed SONET links regularly achieves BERs of < 10 E-15. I
>have
>>> > > > > substantial experience with mainframe serial links such as
>ESCON(tm)
>>> > > > > where the effective system BERs are in the same ballpark. SONET
>uses
>>> > > > > scrambling with long term DC balance and ESCON uses 8B/10B with
>>> short
>>> > > > > term DC balance. The following questions come to mind:
>>> > > > >
>>> > > > > - How important is DC balance?
>>> > > > > - How does this importance scale in going to 10 Gbps?
>>> > > > >
>>> > > > > I'll see if I can get some 8B/10B experts to chime in here if you
>>> can
>>> > > > > get scrambling experts to bear down on the same problem.
>>> > > > >
>>> > > > > --
>>> > > > >
>>> > > > > >(text deleted)
>>> > > > > >
>>> > > > > >The point here is that the SONET scrambler is not the limiting
>>> issue
>>> > > in
>>> > > > > >achieving low error rates. The issue is having enough
>photons/bit,
>>> or
>>> > > > > >optical SNR (eye-Q) to accurately recover the data.
>>> > > > > >
>>> > > > > >...Dave
>>> > > > > >
>>> > > > > >David W. Martin
>>> > > > > >Nortel Networks
>>> > > > > >+1 613 765-2901
>>> > > > > >+1 613 763-2388 (fax)
>>> > > > > >dwmartin@nortelnetworks.com
>>> > > > > >========================
>>> > > >
>>
>>
>
>
>
>
>
Paul A. Bottorff, Director Switching Architecture
Bay Architecture Laboratory
Nortel Networks, Inc.
4401 Great America Parkway
Santa Clara, CA 95052-8185
Tel: 408 495 3365 Fax: 408 495 1299 ESN: 265 3365
email: pbottorf@NortelNetworks.com