Re: Flow control on GigE networks
- To: "bill.st.arnaud" <bill.st.arnaud@canarie.ca>
- Subject: Re: Flow control on GigE networks
- From: Roy Bynum <RBYNUM/0004245935@MCIMAIL.COM>
- Date: Tue, 25 May 1999 16:53:36 -0500 (EST)
- Cc: Drew Perkins <drew.perkins@lightera.com>, "'Andrew Smith'" <andrew@extremenetworks.com>, "'IEEE HSSG'" <stds-802-3-hssg@ieee.org>
- Sender: owner-stds-802-3-hssg@majordomo.ieee.org
Bill,
I understand that 802.3 mapping and transport over SONET/SDH is being
looked at for standardization by more than one organization. Once
802.3 is mapped into fixed portions of the SONET/SDH payload it does
not need flow control at intermediate points. There is more than one
vendor that will be offering this very soon.
One vendor is looking at creating a shared bandwidth version of this.
It would make a SONET/SDH ring act like the back plane of a very large
802.3 switch. It is very interesting to contemplate the implications
of this technology in a WAN environment.
Thank you,
Roy Bynum
Date: Mon May 24, 1999 1:48 pm CST
Source-Date: Mon, 24 May 1999 15:42:34 -0400
Fromm: bill.st.arnaud
EMS: INTERNET / MCI ID: 376-5414
MBX: bill.st.arnaud@canarie.ca
TO: Drew Perkins
EMS: INTERNET / MCI ID: 376-5414
MBX: drew.perkins@lightera.com
TO: * ROY BYNUM / MCI ID: 424-5935
TO: 'Andrew Smith'
EMS: INTERNET / MCI ID: 376-5414
MBX: andrew@extremenetworks.com
CC: 'IEEE HSSG'
EMS: INTERNET / MCI ID: 376-5414
MBX: stds-802-3-hssg@ieee.org
Subject: Flow control on GigE networks
Message-Id: 99052419485795/INTERNETGWDN1IG
Source-Msg-Id: <NBBBJIMEPHPGCNGAHPMFMECBEMAA.bill.st.arnaud@canarie.ca>
U-Importance: Normal
U-X-Mailer: Microsoft Outlook IMO, Build 9.0.2212 (4.71.2419.0)
U-X-MIMEOLE: Produced By Microsoft MimeOLE V5.00.2314.1300
U-X-MSMail-priority: Normal
U-X-Priority: 3 (Normal)
All:
I think there is a dangerous assumption that intermediate GbE regenerative
devices need to do flow control. Many vendors are shipping product that will
do GbE over SONET or transparent optical links many hundreds or thousands of
kilometers with no intermediate flow control.
There are other well know techniques for native Gbe as well.
Bill
-------------------------------------------
Bill St Arnaud
Director Network Projects
CANARIE
bill.st.arnaud@canarie.ca
http://tweetie.canarie.ca/~bstarn
> -----Original Message-----
> From: owner-stds-802-3-hssg@majordomo.ieee.org
> [mailto:owner-stds-802-3-hssg@majordomo.ieee.org]On Behalf Of Drew
> Perkins
> Sent: Thursday, May 20, 1999 2:38 PM
> To: 'Roy Bynum'; 'Andrew Smith'
> Cc: 'IEEE HSSG'
> Subject: RE: A telephony carrier industry perspective
>
>
>
> Having lived through the ATM ABR wars to tell about it, it seems
> to me that
> the Ethernet "link" flow control model is not going to work in this
> environment. That leaves at least two well-known effective approaches.
> Either we reinvent an ATM-style per-VC ABR mechanisms, or we use the
> IP-style RED mechanism. The former would seem to be to be very difficult
> with Ethernet, so I'd vote for the latter.
>
> Drew
> ---------------------------------------------------------
> Ciena Corporation Email: ddp@lightera.com
> Core Switching Division Tel: 408-865-6202
> 10201 Bubb Road Fax: 408-865-6291
> Cupertino, CA 95014 Cell/Pager: 408-829-8298
>
>
> -----Original Message-----
> From: owner-stds-802-3-hssg@majordomo.ieee.org
> [mailto:owner-stds-802-3-hssg@majordomo.ieee.org]On Behalf Of Roy Bynum
> Sent: Wednesday, May 19, 1999 8:14 PM
> To: Andrew Smith
> Cc: IEEE HSSG
> Subject: RE: A telephony carrier industry perspective
>
>
>
> Andrew,
>
> Cascading of congestion flow-control will that will cause the
> degradation of the whole enterprise network because of one circuit,
> will only happen on a network that only has one primary circuit. I
> think that you would be surprised at how fast the "congestion"
> resolves itself in an L2 environment.
>
> I agree with your assessment of the "best effort" IP based Internet.
> Unreliable WAN transport system protocols go all the way back to the
> old BiSync days. What happens when the WAN transport becomes reliable?
> Do you still need the unreliable, best effort protocols? If you had
> a reliable WAN transport at the same cost of an unreliable one, which
> one would you pick?
>
> The TCP flow control at the end systems only works relative to
> retransmissions, which tend to make the congestion worse for longer
> period of time. What happens to mission critical, time critical
> applications in that environment? You don't put those kind of
> applications on an "Internet". This is the reason that carriers like
> MCI WorldCom have companies that out source their data networks to the
> carriers, which in turn provides major revenue. This is the reason
> that data services such as Frame Relay make so much money for
> carriers.
>
> >From a carrier viewpoint, unreliable is unsupportable. I have been
> told by MCI WorldCom's operations and management group, that an
> unreliable 10 GbE will not be deployed as a service in our network. I
> don't know about other carriers, we will stay with reliable SONET
> transport. We can make service level agreements about the reliability
> of the transport on a SONET system. We can not make service level
> agreements on an unreliable 10 GbE. We can do maintenance on SONET
> transport systems without having a major effect on our customers'
> traffic. We can not do that on an unreliable 10 GbE.
>
> Have you ever calculated the amount of data that gets dropped per
> second at a 10Gb rate. Can you imagine the amount of money that a
> carrier would loose because they lost a massive amount of their
> customers' data traffic?
>
> Thank you,
> Roy Bynum
>
>
>
>
>
> Date: Tue May 18, 1999 12:24 pm CST
> Source-Date: Tue, 18 May 1999 11:23:57 -0700
> From: Andrew Smith
> EMS: INTERNET / MCI ID: 376-5414
> MBX: andrew@extremenetworks.com
>
> TO: * ROY BYNUM / MCI ID: 424-5935
> CC: IEEE HSSG
> EMS: INTERNET / MCI ID: 376-5414
> MBX: stds-802-3-hssg@ieee.org
> Subject: RE: A telephony carrier industry perspective
> Message-Id: 99051818240626/INTERNETGWDN3IG
> Source-Msg-Id:
> <D0805D3B448BD211A7990008C7B18130142D09@ftp.extremenetworks.com>
> U-Content-return: allowed
> U-X-Mailer: Internet Mail Service (5.5.2448.0)
>
>
> Roy,
>
> This is very similar territory to that explored recently in the IEEE802
> QoS/Flow-control study group (see
> http://grouper.ieee.org/groups/802/QOSFC/index.html).
>
> Congestion is usually separated into 2 types: short-term, due to
> unforeseen
> traffic peaks, and long-term, due to sustained levels of traffic in excess
> of provisioned capacity. I think you are merging the two types in your
> message. The latter type of congestion is what is usually known as
> "over-subscription" and requires what you term "subscription control".
>
> Queue buffering can be used to handle short-term congestion but is not a
> solution for the second type. It can be used to absorb peaks that occur
> naturally due to well-behaved data flows traversing switching nodes but
> should not be thought of as a good mechanism for handling uncontrolled
> bursty sources. In your scenario, with persistent overload and using
> per-link flow control, "flow-control will continue to cascade back" until
> the whole network becomes equivalent to a shared network segment with an
> *aggregate* bandwidth equivalent to that of the congested link: this is
> likely not going to be popular with your customers.
>
> In the presence of data sources that react to congestion feedback (through
> packet dropping or ECN), the offered load can be throttled at the
> sources to
> match the provisioned capacity: this is how the best-effort Internet works
> today using TCP. There are well-understood packet-dropping mechanisms e.g.
> RED that are commonly implemented these days in routers in the
> Internet core
> that lead to efficient and fair overall network utilisation for
> TCP sources:
> in the Internet architecture, dropping data to signal congestion is
> considered a Good Thing (tm). Uncontrolled (open-loop) data sources are a
> potential problem in such a network and must be controlled by
> shaping at or
> near the source and suitable provisioning if they produce enough
> traffic to
> cause problems.
>
> At the IP layer, mechanisms are being standardised (see
> http://www.ietf.org/html.charters/diffserv-charter.html and RFC 2475 in
> particular) that are widely deployed today in routers for providing
> differential services for different traffic classes on a link and within
> switching nodes: this model assumes the presence of traffic shapers at
> inputs that enforce a per-class contract ("SLA") with the data source that
> can take account of the burstiness of the sources and allows a provider to
> provision appropriately for such pre-agreed peaks without reliance on
> flow-control - in fact, per-link flow control would make it
> difficult for a
> provider to meet such SLAs. The SLAs provide the financial
> controls to make
> this model work.
>
> It is unclear today (to me at least) what added benefit link-layer flow
> control can offer on top of TCP in such networks: any economically viable
> form of flow control is likely to offer, at best, per-class
> control, rather
> than the more desirable per-microflow control. Per-class control penalises
> all microflows that share the class, even those that are well behaved.
> Per-microflow control is expensive and would, in any case,
> duplicate in many
> ways functions that are already provided at the transport layer. Certainly
> there is scope for some mathematical modelling of such scenarios and there
> are also demonstrable scenarios where such detailed flow control is
> worthwhile, particularly those using low-speed, expensive transmission
> lines: I'm not sure that 10G Ethernet falls into that category.
>
> Persistent over-subscription of WAN links for todays enterprise
> networks is
> a solved problem: you either buy a larger SLA or you use bandwidth
> management solutions such as class-based queueing to ensure that the right
> traffic gets through. I really don't hear a lot of people asking for
> link-level flow control to solve this problem.
>
> But we digress ...
>
> Andrew
>
> ****************************************************************
> Andrew Smith tel: +1 (408) 579-2821
> Extreme Networks fax: +1 (408) 579-3000
> 3585 Monroe St. http://www.extremenetworks.com
> Santa Clara CA 95051-1450 em: andrew@extremenetworks.com
> ****************************************************************
>
>
> > -----Original Message-----
> > From: Roy Bynum [mailto:RBYNUM/0004245935@MCIMAIL.COM]
> > Sent: Tuesday, May 18, 1999 7:46 AM
> > To: Andrew Smith
> > Cc: IEEE HSSG
> > Subject: RE: A telephony carrier industry perspective
> >
> >
> > Andrew,
> >
> > I have been designing and implementing all sorts of data networking
> > environments, LAN/MAN/WAN, for some time. One of the headaches is
> > understanding the business application requirements and cost trade-off
> > on link circuit bandwidth. Because intermediate/gateway systems
> > historically do not have the ability to signal back when an output
> > circuit becomes overloaded (the new term is "over-subscribed"), queue
> > buffering has been a major issue. The problem is a network that has
> > circuits that consistently become overloaded and do so for extended
> > lengths of time, such that the queues become overloaded. Data network
> > architects tend to design large enterprise WAN networks based on a
> > consistent loading with an attempt to limit the peaks to close to the
> > maximum circuit bandwidth.
> >
> > With "flow-control" using GbE I can design WAN data networks closer to
> > the consistent loading with less concern for peak traffic. I have set
> > up a test WAN network using Native data GbE over Optical DWDM and
> > SONET DATS to test this. When the L2/L3 GbE switches are properly
> > configured, I can only attempt to overload the network "circuits".
> > When the flow-control is applied all the way to the data source
> > systems, the flow-control will continue to cascade back such that
> > over-subscription does not occur. Unlike a traditional TDM WAN circuit
> > based design, no data is lost during peak "overloading", which tends
> > to prevent more problems because of re-transmissions. As this is
> > better understood and implemented, it will change the bandwidth
> > allocation process of designing enterprise networks.
> >
> > Thank you,
> > Roy Bynum
> >
> >
> >
> > Date: Mon May 17, 1999 5:27 pm CST
> > Source-Date: Mon, 17 May 1999 16:26:58 -0700
> > From: Andrew Smith
> > EMS: INTERNET / MCI ID: 376-5414
> > MBX: andrew@extremenetworks.com
> >
> > TO: * ROY BYNUM / MCI ID: 424-5935
> > CC: IEEE HSSG
> > EMS: INTERNET / MCI ID: 376-5414
> > MBX: stds-802-3-hssg@ieee.org
> > Subject: RE: A telephony carrier industry perspective
> > Message-Id: 99051723270571/INTERNETGWDN1IG
> > Source-Msg-Id:
> > <D0805D3B448BD211A7990008C7B18130142CFC@ftp.extremenetworks.com>
> > U-Content-return: allowed
> > U-X-Mailer: Internet Mail Service (5.5.2448.0)
> >
> > Roy,
> >
> > At the risk of waking some sleeping dogs, could I ask you to
> > elaborate a
> > little on your statement:
> >
> > 'In addition to being less expensive, GbE over Native Data
> > SONET DATS will
> > provide "subscription control" though "flow control" That
> > "subscription
> > control" will prevent over-subscribing the WAN links, which
> > is a big problem
> > for enterprise data network designers and architects.'
> >
> > Andrew
> >
> > ****************************************************************
> > Andrew Smith tel: +1 (408) 579-2821
> > Extreme Networks fax: +1 (408) 579-3000
> > 3585 Monroe St. http://www.extremenetworks.com
> > Santa Clara CA 95051-1450 em: andrew@extremenetworks.com
> > ****************************************************************
> >
> >
> >
> > > -----Original Message-----
> > > From: Roy Bynum [mailto:RBYNUM/0004245935@MCIMAIL.COM]
> > > Sent: Monday, May 17, 1999 4:12 PM
> > > To: bill.st.arnaud
> > > Cc: IEEE HSSG
> > > Subject: RE: A telephony carrier industry perspective
> > >
> > >
> > >
> > > Bill,
> > >
> > > Internet IP will continue to be what the market
> > requirements reflect.
> > > Slow restoration times are acceptable in that environment. The UUNet
> > > Long Haul GbE is part of what I am doing. I am the one that put the
> > > Long Haul Optical switching/Metro DWDM/SONET DATS evaluation of GbE
> > > together.
> > >
> > > As a bit irony, dependable GbE is turning out to be less expensive
> > > than undependable IP over TDM, ATM, or POS. Unless MPLS comes in a
> > > respectable price break, GbE over Native Data SONET DATS
> > will still be
> > > less expensive.
> > >
> > > In addition to being less expensive, GbE over Native Data SONET DATS
> > > will provide "subscription control" though "flow control" That
> > > "subscription control" will prevent over-subscribing the WAN links,
> > > which is a big problem for enterprise data network designers and
> > > architects. Combine that with "priority queueing" and most of what
> > > MPLS was supposed to do has already been accomplished by GbE. You
> > > guys as IEEE have done a greater job than you knew.
> > >
> > > Dependable, high quality transport of Native data traffic
> > such as GbE
> > > and 10GbE is probably going to be a different market, one that "best
> > > effort" is unacceptable to. If there is a market that provides the
> > > profit margin that will sustain 10GbE, it will not be the
> > Internet as
> > > it is today.
> > >
> > > I do know that I have been given the requiement that
> > carriers can not
> > > support a data service over long haul systems that does not provide
> > > "SONET like" functionality. The reason that I joined this
> > study group
> > > is to provide that insight to the standards developers. If
> > that is GbE
> > > or 10GbE over SONET then the issue is already resolved. All that
> > > remains is to determine what the LAN application requirements are,
> > > then the standard can be defined.
> > >
> > >
> > > Date: Mon May 17, 1999 3:23 pm CST
> > > Source-Date: Mon, 17 May 1999 17:17:51 -0400
> > > Fromm: bill.st.arnaud
> > > EMS: INTERNET / MCI ID: 376-5414
> > > MBX: bill.st.arnaud@canarie.ca
> > >
> > > TO: * ROY BYNUM / MCI ID: 424-5935
> > > Subject: RE: A telephony carrier industry perspective
> > > Message-Id: 99051721234042/INTERNETGWDN1IG
> > > Source-Msg-Id:
> > > <NBBBJIMEPHPGCNGAHPMFIEIJELAA.bill.st.arnaud@canarie.ca>
> > > U-Importance: Normal
> > > U-X-Mailer: Microsoft Outlook IMO, Build 9.0.2212 (4.71.2419.0)
> > > U-X-MIMEOLE: Produced By Microsoft MimeOLE V5.00.2314.1300
> > > U-X-MSMail-priority: Normal
> > > U-X-Priority: 3 (Normal)
> > >
> > > Roy:
> > >
> > > Again no disagreement. I don't think traditional SONET or
> > > ATM networks will
> > > disappear. The model we advocate is the same that Frontier is
> > > now deploying:
> > > IP/DWDM for best efforts, slow restoral traffic on one set of
> > > wavelengths,
> > > IP over SONET on another set of wavelengths for those
> > > services that need
> > > fast restoral and security of SONET, and IP over ATM over
> > > SONET on another
> > > set of wavelengths for fine grained QoS services.
> > >
> > > I agree with you that the driving force for GbE is cost. It makes a
> > > dramatic difference to the overall cost of the network.
> > >
> > > But I believe GbE can also make an equal dramatic difference on the
> > > transport side on medium, long haul links up to 1000 km.
> > Your sister
> > > company UUNet has already demonstrated that on some long haul
> > > GbE systems.
> > > But I agree with you this type of link is probably only
> > good for best
> > > efforts IP traffic.
> > >
> > > Bill
> > >
> > > -------------------------------------------
> > > Bill St Arnaud
> > > Director Network Projects
> > > CANARIE
> > > bill.st.arnaud@canarie.ca
> > > http://tweetie.canarie.ca/~bstarn
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Roy Bynum [mailto:RBYNUM/0004245935@MCIMAIL.COM]
> > > > Sent: Monday, May 17, 1999 4:54 PM
> > > > To: bill.st.arnaud
> > > > Cc: IEEE HSSG
> > > > Subject: RE: A telephony carrier industry perspective
> > > >
> > > >
> > > > Bill,
> > > >
> > > > I have an IP base video conferencing demonstration
> > > application that is
> > > > full motion, full resolution. It uses MPEG2 compression and
> > > drives the
> > > > IP data at 7Mbs bidirectional. Part of the demonstration of the
> > > > reliability of GbE over optical and SONET Native Data
> > > systems is to do
> > > > a simulated fiber break. Over a normal IP network, there
> > is a major
> > > > loss of data and thus video synchronization, sometimes
> > the "call" is
> > > > even dropped. Over a Native data network, optical or
> > SONET DATS, if
> > > > you blink, you miss the cut. It is that kind of traffic path
> > > > restoration quality that will be required by future, real time,
> > > > visual, and virtual applications. This is the quality of
> > > traffic path
> > > > restoration that needs to be implemented in Metro/MAN and WAN
> > > > environments. I do not believe that any one protocol and/or
> > > system can
> > > > do that and still be cost effective.
> > > >
> > > > Fromm looking at Cisco's DPT description, it looks a lot like a
> > > > SONET/SDH BLSR ring. In duplicating the alternate path coupled
> > > > architecture of SONET/SDH, Cisco could well duplicate the
> > > restoration
> > > > functionality of SONET/SDH. Although what value it will
> > > have over long
> > > > haul systems that are geared for very high bandwidth, I
> > am not sure.
> > > > Within a LAN environment, that type of restoration is probably not
> > > > required. Within a Metro MAN environment, we have found that GbE
> > > > combined with optical path protected DWDM is very cost effective.
> > > > Cisco will have to come in at a VERY "cheep" cost in order
> > > to justify
> > > > the deployment of their systems. I expect to have some
> > test systems
> > > > before too long. I am looking forward to finding out.
> > > >
> > > > The bottom line to this is cost. Preliminary evaluations
> > are showing
> > > > that GbE already is so cost effective that it is less expensive to
> > > > have 10 GbE interfaces on a router than it is to have 4 OC48 POS
> > > > interfaces. Combine that with the less expensive interfaces on the
> > > > DWDM and SONET DATS equipment it becomes even more attractive. The
> > > > capital cost of IP over "Ethernet" or what I am now calling
> > > Native IP
> > > > is less expensive over a MAN or WAN environment than the
> > > existing TDM
> > > > WAN systems. 10GbE must compete in this environment. In
> > order for it
> > > > to be deployed, it must provide high native data bandwidth,
> > > very cost
> > > > effectively with the specific service, maintenance, and operations
> > > > support that each very different environment requires.
> > > >
> > > >
> > > > Thank you,
> > > > Roy Bynum
> > > >
> > > >
> <much good stuff chopped from below here>
>