Re: Wide Area Networking for the Rest of US - the debate on BER and other issues

[Roy Bynum <rabynum@xxxxxxxxxxx>]

Bill,

Adding a "knob" at the IP layer is the problem of the IETF.  An attempt
has already been made to add an information link between ATM and IP.  A
draft RFC for something call PNNI Aware Routing (PAR) was submitted, but
later pulled.  It seems that the IETF wanted to invent their own layer 2
protocol.  Given that history, I don't think that you will be able to
get what you want.  Without the Optical Networking operations and
mantenance support functions being included in 10GbE, I would not count
on an SNMP MIB register being available to even report bit errors.

					Thank you,
					Roy Bynum



Bill St. Arnaud wrote:
> 
> > Have you ever
> > worked on a large ATM or FR internet that required static, unequial,
> > path costing to be configured on each PVC to prevent OSPF and BGP
> > flapping?
> 
> Yes.  That is why so many of us "netheads" can't wait to get rid of ATM and
> FR have all the knobs and controls at the IP layer using tools like MPLS.
> Similar with BER, the message I have trying to get across is that it should
> be seen as another visible tool at the IP layer and not hidden in the
> transport layer.  We have had too many problems in the past with strange
> interactions between IP and other layers where we can't see the control
> functions or where we can't optimize the IP layer.
> 
> This way we significantly reduce the complexity of the network and ALL the
> knobs and levers that affect network performance are readily available to
> us.
> 
> Bill
> 
>   This type of thing keeps data network designers and
> > implementers awake at night.  Adding additional complexity will just add
> > to the loss of sleep.
> >
> > It would be better to work toward an overall more reliable transport
> > facility.  This should tend to reduce the complexity of designing and
> > implementing WANs.  This should be part of the goal of this group.
> >
> >                                       Thank you,
> >                                       Roy Bynum
> >
> > Bill St. Arnaud wrote:
> > >
> > > Larry:
> > >
> > > What you say may be true for other types of networks.  But
> > dropped packets
> > > and re-transmissions are an essential feature of Internet networks.  The
> > > TCP/IP congestion control mechanisms uses dropped packets as a
> > mechanism to
> > > signal the source to throttle back the data flow.
> > >
> > > In fact many ISPs use a utility called RED ( Random Early
> > Discard ) or WRED
> > > ( Weighted Early Random Discard) to deliberately drop packets
> > as a mechanism
> > > to throttle traffic on congested links.   Yes this does cause a
> > > re-transmission, but TCP automatically drops down to a lower
> > speed when this
> > > happens.  As a result on most Internet links about 1-3% of the
> > traffic is
> > > dropped packet and re=transmissions.  However, most of these
> > dropped packets
> > > are not due to RED but to buffer overflow at the destination receiver.
> > > SIGCOMM'98 has some excellent papers documenting this behaviour on the
> > > Internet.
> > >
> > > If I have to do packet discard in any event I might as well do
> > it a layer 1
> > > just as well as at layer 3.  More importantly if I am already dropping
> > > packets for other reasons, then as long as the number of dropped packets
> > > from BER is less than the number of dropped packets from TCP congestion
> > > control then the actual BER (whether it is 10^-15 or 10^-8) is
> > irrelevant to
> > > me.
> > >
> > > I am assuming that if 10XGbE is used in the long haul the primary
> > > application will be to carry Internet traffic.  That is why it
> > would be nice
> > > to have an option for those of use who are running Internet
> > networks to have
> > > a BER Knob.  With a BER knob I may be able to extend my
> > repeater distance,
> > > use lower cost lasers, etc etc.  However, as I said before this
> > may still
> > > may not be practical because of other issues particularly with
> > respect to
> > > the non-linear factors that affect BER.  But it still might be worth a
> > > cursory investigation.
> > >
> > > Bill
> > >
> > > -------------------------------------------
> > > Bill St Arnaud
> > > Director Network Projects
> > > CANARIE
> > > bill.st.arnaud@xxxxxxxxxx
> > > http://tweetie.canarie.ca/~bstarn
> > >
> > >
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: owner-stds-802-3-hssg@xxxxxxxxxxxxxxxxxx
> > > > [mailto:owner-stds-802-3-hssg@xxxxxxxxxxxxxxxxxx]On Behalf Of Larry
> > > > Miller
> > > > Sent: Wednesday, June 02, 1999 11:38 AM
> > > > To: stds-802-3-hssg@xxxxxxxx
> > > > Subject: Re: Wide Area Networking for the Rest of US - the
> > debate on BER
> > > > and other issues
> > > >
> > > >
> > > >
> > > > I think the bit is that when you report bad frames upward to
> > higher layers
> > > > they have to do some work to re-request those frames and that
> > takes much
> > > > longer than the time actually burned by the dropped frames.
> > Hence, if you
> > > > get too low of a raw BER you spend all (or maybe more than all)
> > > > of your time
> > > > with higher layer thrashing and never get through with the (say) file
> > > > transfer.
> > > >
> > > > This, I think, is the fallacy in Mr St. Arnaud's notion.
> > > >
> > > > Larry Miller
> > > > Nortel Networks
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mike Dudek <mdudek@xxxxxxxxxxxx>
> > > > To: Chang, Edward S <Edward.Chang@xxxxxxxxxx>
> > > > Cc: bin.guo@xxxxxxx <bin.guo@xxxxxxx>; bill.st.arnaud@xxxxxxxxxx
> > > > <bill.st.arnaud@xxxxxxxxxx>; rtaborek@xxxxxxxxxxxxxxxx
> > > > <rtaborek@xxxxxxxxxxxxxxxx>; dwmartin@xxxxxxxxxxxxxxxxxx
> > > > <dwmartin@xxxxxxxxxxxxxxxxxx>; stds-802-3-hssg@xxxxxxxx
> > > > <stds-802-3-hssg@xxxxxxxx>; sachs@xxxxxxxxxxxxxx
> > <sachs@xxxxxxxxxxxxxx>
> > > > Date: Tuesday, June 01, 1999 5:42 PM
> > > > Subject: Re: Wide Area Networking for the Rest of US - the debate
> > > > on BER and
> > > > other issues
> > > >
> > > >
> > > > >
> > > > >Agreed, but the percentage of good frames stays the same.  ie the
> > > > percentage
> > > > >bandwidth used for retransmissions is the same.
> > > > >
> > > > >"Chang, Edward S" wrote:
> > > > >
> > > > >> Mike:
> > > > >>
> > > > >> If the BER is maintained the same for both GbE and 10xGbE
> > and assume
> > > > >> everything is equal, the frequency of getting error from
> > 10GbE is 10
> > > > times
> > > > >> than GbE from PHY.  Of course, the whole system has other
> > factors to be
> > > > >> included to find the final throughput.  In another word,
> > the occurrence
> > > > of
> > > > >> frame error will be much more for 10GbE than GbE.
> > > > >>
> > > > >> I may present mathematical analysis in July, if my time is allowed.
> > > > >>
> > > > >> Ed Chang
> > > > >> Unisys Corporation
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: Mike Dudek [mailto:mdudek@xxxxxxxxxxxx]
> > > > >> Sent: Tuesday, June 01, 1999 10:07 AM
> > > > >> To: Chang, Edward S
> > > > >> Cc: bin.guo@xxxxxxx; bill.st.arnaud@xxxxxxxxxx;
> > > > >> rtaborek@xxxxxxxxxxxxxxxx; dwmartin@xxxxxxxxxxxxxxxxxx;
> > > > >> stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx
> > > > >> Subject: Re: Wide Area Networking for the Rest of US - the
> > > > debate on BER
> > > > >> and other issues
> > > > >>
> > > > >> I do not agree that the BER must be improved with data
> > rate increase in
> > > > >> order to
> > > > >> obtain the higher throughput.  At least for packet based
> > transmission
> > > > with
> > > > >> retransmission of errored packets, the throughput increases in
> > > > proportion
> > > > to
> > > > >> the
> > > > >> data rate for the same BER, assuming that the packet
> > length (in bytes)
> > > > >> remains
> > > > >> fixed.  I do not think that anyone has proposed changing the packet
> > > > length,
> > > > >> but
> > > > >> if they did then the BER might have to be improved.  The
> > > > throughput is of
> > > > >> course
> > > > >> the number of good packets in any interval of time.
> > > > >>
> > > > >> "Chang, Edward S" wrote:
> > > > >>
> > > > >> > Bin:
> > > > >> >
> > > > >> > Yes, I agree.  The BER should be improved with data rate
> > increase, if
> > > > the
> > > > >> > through put gained from higher data rate is to be maintained.  In
> > > > addition
> > > > >> > to the retry times wasted, the external sources of noise
> > remain the
> > > > same,
> > > > >> > which further requires the lower BER.  These are the
> > correct design
> > > > goals
> > > > >> we
> > > > >> > should work on.  Although, we also should keep the
> > cost-effectiveness
> > > > in
> > > > >> > mind to maintain optimum balance between performance and cost.
> > > > >> >
> > > > >> > Ed Chang
> > > > >> > Unisys Corporation
> > > > >> >
> > > > >> > -----Original Message-----
> > > > >> > From: bin.guo@xxxxxxx [mailto:bin.guo@xxxxxxx]
> > > > >> > Sent: Friday, May 28, 1999 4:57 PM
> > > > >> > To: Edward.Chang@xxxxxxxxxx; bill.st.arnaud@xxxxxxxxxx;
> > > > >> > rtaborek@xxxxxxxxxxxxxxxx; dwmartin@xxxxxxxxxxxxxxxxxx
> > > > >> > Cc: stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx;
> > > > "widmer@xxxxxxxxxx
> > > > >> > widmer@xxxxxxxxxx widmer"@us.ibm.com
> > > > >> > Subject: RE: Wide Area Networking for the Rest of US -
> > the debate on
> > > > BER
> > > > >> > a nd other issues
> > > > >> >
> > > > >> > Ed,
> > > > >> >
> > > > >> > If the specified BER for 1000BASE-X is 10^ -12, then to have
> > > > the equal
> > > > >> > error-free period the specified BER for 10G should be at
> > > > least 10^ -13.
> > > > >> > Based on Rich T and Rich S's BER number:
> > > > >> >
> > > > >> > A system BER of 10 E - 8 @  10 Mbps = a bit error every
> > 10 seconds.
> > > > >> > (10BASE-T)
> > > > >> > A system BER of 10 E-12 @ 100 Mbps = a bit error every 166
> > > > minutes, 40
> > > > >> > seconds.        (100BASE-X)
> > > > >> > A system BER of 10 E-10 @     1 Gbps = a bit error every 1
> > > > minutes, 40
> > > > >> > seconds.                (1000BASE-T)
> > > > >> > A system BER of 10 E-12 @     1 Gbps = a bit error every 16
> > > > minutes, 40
> > > > >> > seconds.        (1000BASE-X)
> > > > >> > A system BER of 10 E-12 @   10 Gbps = a bit error every
> > 1 minutes, 40
> > > > >> > seconds.
> > > > >> > A system BER of 10 E-13 @   10 Gbps = a bit error every 16
> > > > minutes, 40
> > > > >> > seconds.
> > > > >> >
> > > > >> > If the TCP/IP is the only protocol 10G PHY needs to
> > support, then the
> > > > >> above
> > > > >> > specified BER may be more than enough.  Moving from 1G to
> > > > 10G, the bit
> > > > >> > period is scaled 10X smaller while jitter and noise from
> > some sources
> > > > are
> > > > >> > not scaled the same way -- much tight control should be
> > applied to
> > > > achieve
> > > > >> > even the same BER.
> > > > >> >
> > > > >> > Bin
> > > > >> >
> > > > >> > ADL,AMD
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > > -----Original Message-----
> > > > >> > > From: Chang, Edward S [SMTP:Edward.Chang@xxxxxxxxxx]
> > > > >> > > Sent: Friday, May 28, 1999 12:44 PM
> > > > >> > > To:   bill.st.arnaud@xxxxxxxxxx; Guo, Bin;
> > > > rtaborek@xxxxxxxxxxxxxxxx;
> > > > >> > > dwmartin@xxxxxxxxxxxxxxxxxx
> > > > >> > > Cc:   stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx;
> > > > "widmer@xxxxxxxxxx
> > > > >> > > widmer@xxxxxxxxxx          widmer"@us.ibm.com
> > > > >> > > Subject:      RE: Wide Area Networking for the Rest of US - the
> > > > debate
> > > > >> on
> > > > >> > > BER a nd other issues
> > > > >> > >
> > > > >> > > Bill:
> > > > >> > >
> > > > >> > > I like your idea of implementing native 10xGBE for
> > > > intermediate long
> > > > >> haul
> > > > >> > > and WAN, which is a good move.  The advantage you are
> > > > mentioning will
> > > > >> > > greatly reduce the cost to users.
> > > > >> > >
> > > > >> > > It is true, in a TCP/IP links, the TCP flow control causes more
> > > > >> > > retransmission than BER. Therefore, the extremely low
> > BER, 10^-15,
> > > > does
> > > > >> > > not
> > > > >> > > necessarily gain any more advantage than the specified BER
> > > > of 10^-12.
> > > > >> > >
> > > > >> > >
> > > > >> > > Ed Chang
> > > > >> > >
> > > > >> > > -----Original Message-----
> > > > >> > > From: Bill St. Arnaud [mailto:bill.st.arnaud@xxxxxxxxxx]
> > > > >> > > Sent: Friday, May 28, 1999 8:52 AM
> > > > >> > > To: bin.guo@xxxxxxx; rtaborek@xxxxxxxxxxxxxxxx;
> > > > >> > > dwmartin@xxxxxxxxxxxxxxxxxx
> > > > >> > > Cc: stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx;
> > > > "widmer@xxxxxxxxxx
> > > > >> > > widmer@xxxxxxxxxx widmer"@us.ibm.com
> > > > >> > > Subject: Wide Area Networking for the Rest of US - the
> > > > debate on BER
> > > > and
> > > > >> > > other issues
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > All:
> > > > >> > > I have been following the interesting debate about BER.
> > > > Let me bring
> > > > >> some
> > > > >> > > further issues into the debate.
> > > > >> > >
> > > > >> > > I am assuming that on WAN and long haul GbE the upper
> > > > layer protocol
> > > > >> will
> > > > >> > > only be IP.
> > > > >> > >
> > > > >> > > On most IP links, even ones with BERs of 10^-15 there
> > is about 1-3%
> > > > >> packet
> > > > >> > > loss and retransmission.  This is due to a number of
> > > > factors but most
> > > > >> > > typically it relates to TCP flow control mechanism from
> > > > server bound
> > > > >> > > congestion (not network congestion) and the use of
> > WRED in routers.
> > > > >> > >
> > > > >> > > So, on most IP links the packet loss due to BER is
> > > > significantly less
> > > > >> than
> > > > >> > > that due to normal TCP congestion.  As long as that ratio is
> > > > maintained
> > > > >> it
> > > > >> > > is largely irrelevant what the absolute BER value is.
> > > > There will be
> > > > >> many
> > > > >> > > more retransmissions from the IP layer than there will
> > be at the
> > > > >> physical
> > > > >> > > layer due to BER.
> > > > >> > >
> > > > >> > > Other protocols like Frame Relay and SNA are a lot more
> > > > sensitive to
> > > > >> high
> > > > >> > > BERs.  IP ( in particular TCP/IP) is significantly
> > more robust and
> > > > can
> > > > >> > > work
> > > > >> > > quite effectively in high BER environments e.g. TCP/IP
> > over barbed
> > > > wire.
> > > > >> > >
> > > > >> > > I would like to suggest that the 802.3 HSSG group consider an 2
> > > > >> solutions
> > > > >> > > for 10xGbE WAN:
> > > > >> > > (1) native 10xGbE using 8b/10b; and
> > > > >> > > (2)10xGbE mapped to a SONET STS OC-192 frame
> > > > >> > >
> > > > >> > > For extreme long haul solutions SONET makes a lot of sense as a
> > > > >> transport
> > > > >> > > technology.  However for intermediate long haul (up to
> > 1000 km) and
> > > > WAN
> > > > >> > > native 10xGbE is more attractive. Native GbE can be either
> > > > transported
> > > > >> on
> > > > >> > > a
> > > > >> > > transparent optical network or carried directly on a CWDM
> > > > system with
> > > > >> > > transceivers. In medium range networks coding
> > efficiency is not as
> > > > >> > > important
> > > > >> > > as it is in long haul networks. If coding efficiency
> > is important
> > > > then
> > > > >> in
> > > > >> > > my
> > > > >> > > opinion, it does not make sense to invent a new coding
> > scheme for
> > > > 10xGbE
> > > > >> > > when it would be just as easy to map it to a SONET frame.
> > > > >> > >
> > > > >> > > The attraction of native 10xGbE for the WAN is that it
> > is a "wide
> > > > area
> > > > >> > > networking solution for the rest of us".  You don't
> > need to hire
> > > > >> > > specialized
> > > > >> > > SONET engineers to run and manage your networks.  The 18
> > > > year old kid
> > > > >> who
> > > > >> > > is
> > > > >> > > running your LAN can now easily learn to operate and
> > manage a WAN.
> > > > >> > >
> > > > >> > > In Canada and the US, there are several vendors who
> > are willing to
> > > > sell
> > > > >> > > dark
> > > > >> > > fiber at a very reasonable cost.  Right now the cost
> > of building a
> > > > WAN
> > > > >> > > with
> > > > >> > > 10xGbE and CWDM is substantially less (for comparable
> > data rates)
> > > > than
> > > > >> > > using
> > > > >> > > SONET equipment.
> > > > >> > >
> > > > >> > > Bill
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > -------------------------------------------
> > > > >> > > Bill St Arnaud
> > > > >> > > Director Network Projects
> > > > >> > > CANARIE
> > > > >> > > bill.st.arnaud@xxxxxxxxxx
> > > > >> > > http://tweetie.canarie.ca/~bstarn
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > > -----Original Message-----
> > > > >> > > > From: owner-stds-802-3-hssg@xxxxxxxxxxxxxxxxxx
> > > > >> > > > [mailto:owner-stds-802-3-hssg@xxxxxxxxxxxxxxxxxx]On Behalf Of
> > > > >> > > > bin.guo@xxxxxxx
> > > > >> > > > Sent: Thursday, May 27, 1999 7:28 PM
> > > > >> > > > To: rtaborek@xxxxxxxxxxxxxxxx; dwmartin@xxxxxxxxxxxxxxxxxx
> > > > >> > > > Cc: stds-802-3-hssg@xxxxxxxx; sachs@xxxxxxxxxxxxxx;
> > > > "widmer@xxxxxxxxxx
> > > > >> > > > widmer@xxxxxxxxxx widmer"@us.ibm.com
> > > > >> > > > Subject: RE: 1000BASE-T PCS question
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Rich,
> > > > >> > > >
> > > > >> > > > The DC balance can be directly translated into jitter
> > > > (when timing
> > > > is
> > > > >> > > > concerned) and offset (when threshold slicing is
> > concerned).  You
> > > > >> > > > only need
> > > > >> > > > to deal with the former if the signal is 2-level
> > NRZI, while you
> > > > need
> > > > >> to
> > > > >> > > > deal with both if multi-level signal modulation is used.
> > > > >> > > >
> > > > >> > > > For long term DC imbalance, it translates into low
> > > > frequency jitter
> > > > >> and
> > > > >> > > if
> > > > >> > > > it's low enough(<1 KHz ?), it's called baseline wonder.  For
> > > > >> > > > short term, it
> > > > >> > > > relates to Data Dependent Jitter, which is more difficult for
> > > > timing
> > > > >> > > > recovery to handle since it's not from system or channel
> > > > imparity,
> > > > and
> > > > >> > > > therefore it's harder to compensate.
> > > > >> > > >
> > > > >> > > > When you have a lot of jitter margin, for example in
> > lower speed
> > > > >> > > clocking,
> > > > >> > > > the amount of jitter, translated from DC drift
> > resulted from data
> > > > >> > > > imbalance
> > > > >> > > > coupled by AC circuit, percentage wise is a small
> > portion of the
> > > > clock
> > > > >> > > > period and therefore does not contribute to much of the eye
> > > > >> > > > closing.  On the
> > > > >> > > > other hand, for high speed clocking at 10G (100
> > ps?), the jitter
> > > > >> > > > translated
> > > > >> > > > from the same amount of DC drift can be a
> > significant portion of
> > > > the
> > > > >> > > clock
> > > > >> > > > period, so contributes to much large percentage wise
> > jitter which
> > > > >> > > > results in
> > > > >> > > > reduced eye opening -- higher BER.
> > > > >> > > >
> > > > >> > > > Dave said in his mail that "The limiting factor is enough RX
> > > > optical
> > > > >> > > power
> > > > >> > > > to provide a sufficiently open eye." but you still
> > have to deal
> > > > with
> > > > >> the
> > > > >> > > > data dependent jitter due to DC imbalance generated
> > > > after O/E, that
> > > > >> can
> > > > >> > > > close the eye further again.
> > > > >> > > >
> > > > >> > > > Bin
> > > > >> > > >
> > > > >> > > > ADL, AMD
> > > > >> > > >
> > > > >> > > > > -----Original Message-----
> > > > >> > > > > From:     Rich Taborek [SMTP:rtaborek@xxxxxxxxxxxxxxxx]
> > > > >> > > > > Sent:     Thursday, May 27, 1999 3:23 PM
> > > > >> > > > > To:       David Martin
> > > > >> > > > > Cc:       HSSG_reflector; Sachs,Marty; Widmer,Albert_X
> > > > >> > > > > Subject:  Re: 1000BASE-T PCS question
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Dave,
> > > > >> > > > >
> > > > >> > > > > Do you know of any research or other proofs in this
> > > > area? You say
> > > > >> that
> > > > >> > > > > lower speed SONET links regularly achieves BERs of
> > < 10 E-15. I
> > > > have
> > > > >> > > > > substantial experience with mainframe serial links such as
> > > > ESCON(tm)
> > > > >> > > > > where the effective system BERs are in the same
> > ballpark. SONET
> > > > uses
> > > > >> > > > > scrambling with long term DC balance and ESCON
> > uses 8B/10B with
> > > > >> short
> > > > >> > > > > term DC balance. The following questions come to mind:
> > > > >> > > > >
> > > > >> > > > > - How important is DC balance?
> > > > >> > > > > - How does this importance scale in going to 10 Gbps?
> > > > >> > > > >
> > > > >> > > > > I'll see if I can get some 8B/10B experts to chime in
> > > > here if you
> > > > >> can
> > > > >> > > > > get scrambling experts to bear down on the same problem.
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > >
> > > > >> > > > > >(text deleted)
> > > > >> > > > > >
> > > > >> > > > > >The point here is that the SONET scrambler is not
> > the limiting
> > > > >> issue
> > > > >> > > in
> > > > >> > > > > >achieving low error rates. The issue is having enough
> > > > photons/bit,
> > > > >> or
> > > > >> > > > > >optical SNR (eye-Q) to accurately recover the data.
> > > > >> > > > > >
> > > > >> > > > > >...Dave
> > > > >> > > > > >
> > > > >> > > > > >David W. Martin
> > > > >> > > > > >Nortel Networks
> > > > >> > > > > >+1 613 765-2901
> > > > >> > > > > >+1 613 763-2388 (fax)
> > > > >> > > > > >dwmartin@xxxxxxxxxxxxxxxxxx
> > > > >> > > > > >========================
> > > > >> > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> >